Towards A Unified Agent with Foundation Models

18 July 2023

Markus Wulfmeier

Martin Riedmiller

Papers citing "Towards A Unified Agent with Foundation Models"

45 / 45 papers shown

Title
LLM as GNN: Graph Vocabulary Learning for Text-Attributed Graph Foundation Models Xi Zhu Haochen Xue Ziwei Zhao Wujiang Xu Jingyuan Huang Minghao Guo Qifan Wang Kaixiong Zhou Yongfeng Zhang 53 1 0 05 Mar 2025
A Large Recurrent Action Model: xLSTM enables Fast Inference for Robotics Tasks Thomas Schmied Thomas Adler Vihang Patil M. Beck Korbinian Poppel Johannes Brandstetter G. Klambauer Razvan Pascanu Sepp Hochreiter 70 4 0 21 Feb 2025
Scaling Autonomous Agents via Automatic Reward Modeling And Planning Zhenfang Chen Delin Chen Rui Sun Wenjun Liu Chuang Gan LLMAG 58 3 0 17 Feb 2025
Interacting Large Language Model Agents. Interpretable Models and Social Learning Adit Jain Vikram Krishnamurthy LLMAG 28 0 0 02 Nov 2024
VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AI Sijie Cheng Kechen Fang Yangyang Yu Sicheng Zhou B. Li Ye Tian Tingguang Li Lei Han Yang Janet Liu 37 8 0 15 Oct 2024
The Ability of Large Language Models to Evaluate Constraint-satisfaction in Agent Responses to Open-ended Requests Lior Madmoni Amir Zait Ilia Labzovsky Danny Karmon ELM 28 0 0 22 Sep 2024
NEUSIS: A Compositional Neuro-Symbolic Framework for Autonomous Perception, Reasoning, and Planning in Complex UAV Search Missions Zhixi Cai Cristian Rojas Cardenas Kevin Leo Chenyuan Zhang Kal Backman ... Yuan-Fang Li Mor Vered Peter James Stuckey M. G. D. L. Banda Hamid Rezatofighi 29 5 0 16 Sep 2024
Game On: Towards Language Models as RL Experimenters Jingwei Zhang Thomas Lampe A. Abdolmaleki Jost Tobias Springenberg Martin Riedmiller LM&Ro 29 0 0 05 Sep 2024
Diffusion Augmented Agents: A Framework for Efficient Exploration and Transfer Learning Norman Di Palo Leonard Hasenclever Jan Humplik Arunkumar Byravan 16 2 0 30 Jul 2024
MaxMI: A Maximal Mutual Information Criterion for Manipulation Concept Discovery Pei Zhou Yanchao Yang 27 1 0 21 Jul 2024
MaPPing Your Model: Assessing the Impact of Adversarial Attacks on LLM-based Programming Assistants John Heibel Daniel Lowd AAML 24 3 0 12 Jul 2024
Improving Sample Efficiency of Reinforcement Learning with Background Knowledge from Large Language Models Fuxiang Zhang Junyou Li Yi-Chen Li Zongzhang Zhang Yang Yu Deheng Ye OffRL KELM 39 1 0 04 Jul 2024
A Survey of Language-Based Communication in Robotics William Hunt Sarvapali D. Ramchurn Mohammad D. Soorati LM&Ro 47 12 0 06 Jun 2024
Automating the Enterprise with Foundation Models Michael Wornow A. Narayan Krista Opsahl-Ong Quinn McIntyre Nigam H. Shah Christopher Ré AI4CE 20 9 0 03 May 2024
Overcoming Knowledge Barriers: Online Imitation Learning from Visual Observation with Pretrained World Models Xingyuan Zhang Philip Becker-Ehmck Patrick van der Smagt Maximilian Karl OffRL 42 0 0 29 Apr 2024
What Foundation Models can Bring for Robot Learning in Manipulation : A Survey Dingzhe Li Yixiang Jin A. Yong Hongze Yu Jun Shi Xiaoshuai Hao Peng Hao Huaping Liu Fuchun Sun Bin Fang AI4CE LM&Ro 64 12 0 28 Apr 2024
InfoCon: Concept Discovery with Generative and Discriminative Informativeness Ruizhe Liu Qian Luo Yanchao Yang 27 2 0 14 Mar 2024
ExploRLLM: Guiding Exploration in Reinforcement Learning with Large Language Models Runyu Ma Jelle Luijkx Zlatan Ajanović Jens Kober LM&Ro LRM 33 7 0 14 Mar 2024
VANP: Learning Where to See for Navigation with Self-Supervised Vision-Action Pre-Training Mohammad Nazeri Junzhe Wang Amirreza Payandeh Xuesu Xiao SSL ViT 41 5 0 12 Mar 2024
Embodied Understanding of Driving Scenarios Yunsong Zhou Linyan Huang Qingwen Bu Jia Zeng Tianyu Li Hang Qiu Hongzi Zhu Minyi Guo Yu Qiao Hongyang Li LM&Ro 55 30 0 07 Mar 2024
Bootstrapping Cognitive Agents with a Large Language Model Feiyu Zhu Reid Simmons LLMAG 27 6 0 25 Feb 2024
BBSEA: An Exploration of Brain-Body Synchronization for Embodied Agents Sizhe Yang Qian Luo Anumpam Pani Yanchao Yang 22 2 0 13 Feb 2024
Vision-Language Models Provide Promptable Representations for Reinforcement Learning William Chen Oier Mees Aviral Kumar Sergey Levine VLM LM&Ro 39 21 0 05 Feb 2024
Hierarchical Continual Reinforcement Learning via Large Language Model Chaofan Pan Xin Yang Hao Wang Wei Wei Tianrui Li 13 2 0 25 Jan 2024
Large Language Models for Robotics: Opportunities, Challenges, and Perspectives Jiaqi Wang Zihao Wu Yiwei Li Hanqi Jiang Peng Shu ... Lin Zhao Bao Ge Xiang Li Tianming Liu Shu Zhang LM&Ro 27 60 0 09 Jan 2024
Foundation Models in Robotics: Applications, Challenges, and the Future Roya Firoozi Johnathan Tucker Stephen Tian Anirudha Majumdar Jiankai Sun ... Brian Ichter Danny Driess Jiajun Wu Cewu Lu Mac Schwager LM&Ro AI4CE LRM VLM 35 136 0 13 Dec 2023
KwaiAgents: Generalized Information-seeking Agent System with Large Language Models Haojie Pan Zepeng Zhai Hao Yuan Yaojia Lv Ruiji Fu Ming Liu Zhongyuan Wang Bing Qin LLMAG RALM 13 9 0 08 Dec 2023
Kinematic-aware Prompting for Generalizable Articulated Object Manipulation with LLMs Wenke Xia Dong Wang Xincheng Pang Zhigang Wang Bin Zhao Di Hu Xuelong Li LM&Ro 27 19 0 06 Nov 2023
Language Models as Zero-Shot Trajectory Generators Teyun Kwon Norman Di Palo Edward Johns LM&Ro 25 44 0 17 Oct 2023
GROOT: Learning to Follow Instructions by Watching Gameplay Videos Shaofei Cai Bowei Zhang Zihao Wang Xiaojian Ma Anji Liu Yitao Liang 83 26 0 12 Oct 2023
Motif: Intrinsic Motivation from Artificial Intelligence Feedback Martin Klissarov P. DÓro Shagun Sodhani Roberta Raileanu Pierre-Luc Bacon Pascal Vincent Amy Zhang Mikael Henaff LRM LLMAG 16 54 0 29 Sep 2023
Prompt, Plan, Perform: LLM-based Humanoid Control via Quantized Imitation Learning Jingkai Sun Qiang Zhang Yiqun Duan Xiaoyang Jiang Chong Cheng Renjing Xu LM&Ro 35 22 0 20 Sep 2023
Are Large Language Models Really Robust to Word-Level Perturbations? Haoyu Wang Guozheng Ma Cong Yu Ning Gui Linrui Zhang ... Sen Zhang Li Shen Xueqian Wang Peilin Zhao Dacheng Tao KELM 21 22 0 20 Sep 2023
Cognitive Architectures for Language Agents T. Sumers Shunyu Yao Karthik Narasimhan Thomas L. Griffiths LLMAG LM&Ro 34 150 0 05 Sep 2023
Expanding Frozen Vision-Language Models without Retraining: Towards Improved Robot Perception Riley Tavassoli Mani Amani Reza Akhavian 22 1 0 31 Aug 2023
A Survey on Large Language Model based Autonomous Agents Lei Wang Chengbang Ma Xueyang Feng Zeyu Zhang Hao-ran Yang ... Xu Chen Yankai Lin Wayne Xin Zhao Zhewei Wei Ji-Rong Wen LLMAG AI4CE LM&Ro 39 1,101 0 22 Aug 2023
Language is All a Graph Needs Ruosong Ye Caiqi Zhang Runhui Wang Shuyuan Xu Yongfeng Zhang AI4CE 45 150 0 14 Aug 2023
LLM as A Robotic Brain: Unifying Egocentric Memory and Control Jinjie Mai Jun Chen Bing Li Guocheng Qian Mohamed Elhoseiny Bernard Ghanem LM&Ro 12 33 0 19 Apr 2023
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models Junnan Li Dongxu Li Silvio Savarese Steven C. H. Hoi VLM MLLM 247 4,186 0 30 Jan 2023
In-context Learning and Induction Heads Catherine Olsson Nelson Elhage Neel Nanda Nicholas Joseph Nova Dassarma ... Tom B. Brown Jack Clark Jared Kaplan Sam McCandlish C. Olah 240 453 0 24 Sep 2022
MineRL Diamond 2021 Competition: Overview, Results, and Lessons Learned Anssi Kanervisto Stephanie Milani Karolis Ramanauskas Nicholay Topin Zichuan Lin ... Franccois Fleuret Alexander Nikulin Yury Belousov Oleg Svidchenko A. Shpilman OffRL 47 31 0 17 Feb 2022
Masked Autoencoders Are Scalable Vision Learners Kaiming He Xinlei Chen Saining Xie Yanghao Li Piotr Dollár Ross B. Girshick ViT TPM 258 7,337 0 11 Nov 2021
Is Curiosity All You Need? On the Utility of Emergent Behaviours from Curious Exploration Oliver Groth Markus Wulfmeier Giulia Vezzani Vibhavari Dasagi Tim Hertweck Roland Hafner N. Heess Martin Riedmiller LRM 31 20 0 17 Sep 2021
Emerging Properties in Self-Supervised Vision Transformers Mathilde Caron Hugo Touvron Ishan Misra Hervé Jégou Julien Mairal Piotr Bojanowski Armand Joulin 286 5,723 0 29 Apr 2021
Model-Based Inverse Reinforcement Learning from Visual Demonstrations Neha Das Sarah Bechtle Todor Davchev Dinesh Jayaraman Akshara Rai Franziska Meier 111 82 0 18 Oct 2020