MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge

17 June 2022

Linxi Fan

De-An Huang

Papers citing "MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge"

50 / 286 papers shown

Title
CLSP: High-Fidelity Contrastive Language-State Pre-training for Agent State Representation Fuxian Huang Qi Zhang Shaopeng Zhai Jie Wang Tianyi Zhang Haoran Zhang Ming Zhou Yu Liu Yu Qiao CLIP AI4TS 34 0 0 24 Sep 2024
Synatra: Turning Indirect Knowledge into Direct Demonstrations for Digital Agents at Scale Tianyue Ou Frank F. Xu Aman Madaan J. Liu Robert Lo Abishek Sridhar Sudipta Sengupta Dan Roth Graham Neubig Shuyan Zhou OffRL 25 9 0 24 Sep 2024
A Survey on Multimodal Benchmarks: In the Era of Large AI Models Lin Li Guikun Chen Hanrong Shi Jun Xiao Long Chen 34 9 0 21 Sep 2024
LASP: Surveying the State-of-the-Art in Large Language Model-Assisted AI Planning Haoming Li Zhaoliang Chen Jonathan Zhang Fei Liu LM&Ro LLMAG LRM 39 4 0 03 Sep 2024
EPO: Hierarchical LLM Agents with Environment Preference Optimization Qi Zhao Haotian Fu Chen Sun G. Konidaris 20 7 0 28 Aug 2024
VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agents Xiao-Yang Liu Tianjie Zhang Yu Gu Iat Long Iong Yifan Xu ... Zhengxiao Du Chan Hee Song Yu Su Yuxiao Dong Jie Tang VLM LLMAG 36 22 0 12 Aug 2024
Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Tasks Zaijing Li Yuquan Xie Rui Shao Gongwei Chen Dongmei Jiang Liqiang Nie 49 18 0 07 Aug 2024
Visual Grounding for Object-Level Generalization in Reinforcement Learning Haobin Jiang Zongqing Lu LM&Ro 25 2 0 04 Aug 2024
PianoMime: Learning a Generalist, Dexterous Piano Player from Internet Demonstrations Cheng Qian Julen Urain Kevin Zakka Jan Peters 17 4 0 25 Jul 2024
Adapt2Reward: Adapting Video-Language Models to Generalizable Robotic Rewards via Failure Prompts Yanting Yang Minghao Chen Qibo Qiu Jiahao Wu Wenxiao Wang Binbin Lin Ziyu Guan Xiaofei He LM&Ro 32 2 0 20 Jul 2024
GRUtopia: Dream General Robots in a City at Scale Hanqing Wang Jiahe Chen Wensi Huang Qingwei Ben Tai Wang ... Ying Zhao Zhongying Tu Yu Qiao Dahua Lin Jiangmiao Pang LM&Ro VGen 47 16 0 15 Jul 2024
Instruction Following with Goal-Conditioned Reinforcement Learning in Virtual Environments Zoya Volovikova A. Skrynnik Petr Kuderov Aleksandr I. Panov LLMAG LM&Ro 22 0 0 12 Jul 2024
IDAT: A Multi-Modal Dataset and Toolkit for Building and Evaluating Interactive Task-Solving Agents Shrestha Mohanty Negar Arabzadeh Andrea Tupini Yuxuan Sun Alexey Skrynnik Artem Zholus Marc-Alexandre Côté Julia Kiseleva 33 0 0 12 Jul 2024
Richelieu: Self-Evolving LLM-Based Agents for AI Diplomacy Zhenyu Guan Xiangyu Kong Fangwei Zhong Yizhou Wang 31 4 0 09 Jul 2024
Craftium: An Extensible Framework for Creating Reinforcement Learning Environments Mikel Malagón Josu Ceberio Jose A. Lozano 27 0 0 04 Jul 2024
Multi-State-Action Tokenisation in Decision Transformers for Multi-Discrete Action Spaces Perusha Moodley Pramod S. Kaushik Dhillu Thambi Mark Trovinger Praveen Paruchuri Xia Hong Benjamin Rosman 42 0 0 01 Jul 2024
Disentangled Representations for Causal Cognition Filippo Torresan Manuel Baltieri CML 27 1 0 30 Jun 2024
OmniJARVIS: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agents Zihao Wang Shaofei Cai Zhancun Mu Haowei Lin Ceyao Zhang Xuejie Liu Qing Li Anji Liu Xiaojian Ma Yitao Liang LM&Ro 30 11 0 27 Jun 2024
Multimodal foundation world models for generalist embodied agents Pietro Mazzaglia Tim Verbelen Bart Dhoedt Aaron C. Courville Sai Rajeswar OffRL LM&Ro 37 1 0 26 Jun 2024
DKPROMPT: Domain Knowledge Prompting Vision-Language Models for Open-World Planning Xiaohan Zhang Zainab Altaweel Yohei Hayamizu Yan Ding S. Amiri Hao Yang Andy Kaminski Chad Esselink Shiqi Zhang VLM LM&Ro 31 6 0 25 Jun 2024
QuadrupedGPT: Towards a Versatile Quadruped Agent in Open-ended Worlds Ye Wang Yuting Mei Sipeng Zheng Qin Jin LRM 27 2 0 24 Jun 2024
LangSuitE: Planning, Controlling and Interacting with Large Language Models in Embodied Text Environments Zixia Jia Mengmeng Wang Baichen Tong Song-Chun Zhu Zilong Zheng LM&Ro LLMAG 34 4 0 24 Jun 2024
Do Multimodal Foundation Models Understand Enterprise Workflows? A Benchmark for Business Process Management Tasks Michael Wornow A. Narayan Ben T Viggiano Ishan S. Khare Tathagat Verma ... Joshua Martinez Vardhan Agrawal Althea Hudson N. Shah Christopher Ré 35 4 0 19 Jun 2024
STEVE Series: Step-by-Step Construction of Agent Systems in Minecraft Zhonghan Zhao Wenhao Chai Xuan Wang Ke Ma Kewei Chen Dongxu Guo Tian Ye Yanting Zhang Hongwei Wang Gaoang Wang LLMAG LM&Ro 18 6 0 17 Jun 2024
DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents Peter Alexander Jansen Marc-Alexandre Côté Tushar Khot Erin Bransom Bhavana Dalvi Mishra Bodhisattwa Prasad Majumder Oyvind Tafjord Peter Clark LLMAG 35 21 0 10 Jun 2024
Online Adaptation for Enhancing Imitation Learning Policies Federico Malato Ville Hautamaki OnRL 16 1 0 07 Jun 2024
AgentGym: Evolving Large Language Model-based Agents across Diverse Environments Zhiheng Xi Yiwen Ding Wenxiang Chen Boyang Hong Honglin Guo ... Qi Zhang Xipeng Qiu Xuanjing Huang Zuxuan Wu Yu-Gang Jiang LLMAG LM&Ro 38 28 0 06 Jun 2024
The Embodied World Model Based on LLM with Visual Information and Prediction-Oriented Prompts Wakana Haijima Kou Nakakubo Masahiro Suzuki Yutaka Matsuo 28 1 0 02 Jun 2024
Video-Language Critic: Transferable Reward Functions for Language-Conditioned Robotics Minttu Alakuijala Reginald McLean Isaac Woungang Nariman Farsad Samuel Kaski Pekka Marttinen Kai Yuan LM&Ro 29 0 0 30 May 2024
Position: Foundation Agents as the Paradigm Shift for Decision Making Xiaoqian Liu Xingzhou Lou Jianbin Jiao Junge Zhang OffRL LLMAG 31 5 0 27 May 2024
LARM: Large Auto-Regressive Model for Long-Horizon Embodied Intelligence Zhuoling Li Xiaogang Xu Zhenhua Xu Sernam Lim Hengshuang Zhao LM&Ro 34 2 0 27 May 2024
Luban: Building Open-Ended Creative Agents via Autonomous Embodied Verification Yuxuan Guo Shaohui Peng Jiaming Guo Di Huang Xishan Zhang ... Zihao Zhang Zidong Du Qi Guo Xingui Hu Yunji Chen 24 4 0 24 May 2024
TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correction Yunfan Jiang Chen Wang Ruohan Zhang Jiajun Wu Fei-Fei Li OnRL 30 25 0 16 May 2024
Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning Yuexiang Zhai Hao Bai Zipeng Lin Jiayi Pan Shengbang Tong ... Alane Suhr Saining Xie Yann LeCun Yi-An Ma Sergey Levine LLMAG LRM 29 54 0 16 May 2024
Natural Language Can Help Bridge the Sim2Real Gap Albert Yu Adeline Foote Raymond J. Mooney Roberto Martín-Martín LM&Ro 30 11 0 16 May 2024
Imitation Learning: A Survey of Learning Methods, Environments and Metrics Nathan Gavenski Odinaldo Rodrigues Michael Luck 24 167 0 30 Apr 2024
MMAC-Copilot: Multi-modal Agent Collaboration Operating Copilot Zirui Song Yaohang Li Meng Fang Zhenhao Chen Zecheng Shi Yuan Huang Ling-Hao Chen Xiuying Chen Ling Chen LLMAG 29 1 0 28 Apr 2024
Ag2Manip: Learning Novel Manipulation Skills with Agent-Agnostic Visual and Action Representations Puhao Li Tengyu Liu Yuyang Li Muzhi Han Haoran Geng Shu Wang Yixin Zhu Song-Chun Zhu Siyuan Huang 34 16 0 26 Apr 2024
Benchmarking Mobile Device Control Agents across Diverse Configurations Juyong Lee Taywon Min Minyong An Changyeon Kim Kimin Lee 25 8 0 25 Apr 2024
DreamCraft: Text-Guided Generation of Functional 3D Environments in Minecraft Sam Earle Filippos Kokkinos Yuhe Nie Julian Togelius Roberta Raileanu 19 8 0 23 Apr 2024
Do We Really Need a Complex Agent System? Distill Embodied Agent into a Single Model Zhonghan Zhao Ke Ma Wenhao Chai Xuan Wang Kewei Chen Dongxu Guo Yanting Zhang Hongwei Wang Gaoang Wang 32 13 0 06 Apr 2024
Evolving Agents: Interactive Simulation of Dynamic and Diverse Human Personalities Jiale Li Jiayang Li Jiahao Chen Yifan Li Shijie Wang Hugo Zhou Minjun Ye Yunsheng Su AI4CE 40 4 0 03 Apr 2024
A Survey on Large Language Model-Based Game Agents Sihao Hu Tiansheng Huang Gaowen Liu Ramana Rao Kompella Gaowen Liu Selim Furkan Tekin Yichang Xu Zachary Yahn Ling Liu LLMAG LM&Ro AI4CE LM&MA 66 49 0 02 Apr 2024
Instruction-Driven Game Engines on Large Language Models Hongqiu Wu Xing-Chen Liu Haizhen Zhao Min Zhang 32 1 0 30 Mar 2024
MANGO: A Benchmark for Evaluating Mapping and Navigation Abilities of Large Language Models Peng Ding Jiading Fang Peng Li Kangrui Wang Xiaochen Zhou Mo Yu Jing Li Matthew R. Walter Hongyuan Mei RALM ELM 40 5 0 29 Mar 2024
MineLand: Simulating Large-Scale Multi-Agent Interactions with Limited Multimodal Senses and Physical Needs Xianhao Yu Jiaqi Fu Renjia Deng Wenjuan Han 26 5 0 28 Mar 2024
Untangling Knots: Leveraging LLM for Error Resolution in Computational Notebooks Konstantin Grotov Sergey Titov Yaroslav Zharov T. Bryksin 21 0 0 26 Mar 2024
AIOS: LLM Agent Operating System Kai Mei Zelong Li Wujiang Xu Wenyue Hua Mingyu Jin Yongfeng Zhang Shuyuan Xu Ruosong Ye Yingqiang Ge Yongfeng Zhang LLMAG 26 17 0 25 Mar 2024
Temporal-Spatial Object Relations Modeling for Vision-and-Language Navigation Bowen Huang Yanwei Zheng Chuanlin Lan Xinpeng Zhao Yifei Zou Dongxiao Yu 31 0 0 23 Mar 2024
MineDreamer: Learning to Follow Instructions via Chain-of-Imagination for Simulated-World Control Enshen Zhou Yiran Qin Zhen-fei Yin Yuzhou Huang Ruimao Zhang Lu Sheng Yu Qiao Jing Shao LM&Ro AI4CE 37 32 0 18 Mar 2024