MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge

17 June 2022

Linxi Fan

De-An Huang

Papers citing "MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge"

50 / 286 papers shown

Title
Visual Encoders for Data-Efficient Imitation Learning in Modern Video Games Lukas Schäfer Logan Jones Anssi Kanervisto Yuhan Cao Tabish Rashid Raluca Georgescu David Bignell Siddhartha Sen Andrea Trevino Gavito Sam Devlin 82 3 0 04 Dec 2023
Planning as In-Painting: A Diffusion-Based Embodied Task Planning Framework for Environments under Uncertainty Cheng-Fu Yang Haoyang Xu Te-Lin Wu Xiaofeng Gao Kai-Wei Chang Feng Gao DiffM 19 8 0 02 Dec 2023
Deciphering Digital Detectives: Understanding LLM Behaviors and Capabilities in Multi-Agent Mystery Games Dekun Wu Haochen Shi Zhiyuan Sun Bang Liu LLMAG 13 15 0 01 Dec 2023
See and Think: Embodied Agent in Virtual Environment Zhonghan Zhao Wenhao Chai Xuan Wang Li Boyi Shengyu Hao Shidong Cao Tianbo Ye Gaoang Wang LM&Ro LLMAG 13 31 0 26 Nov 2023
Robot Learning in the Era of Foundation Models: A Survey Xuan Xiao Jiahang Liu Zhipeng Wang Yanmin Zhou Yong Qi Qian Cheng Bin He Shuo Jiang AI4CE LM&Ro 16 25 0 24 Nov 2023
An Embodied Generalist Agent in 3D World Jiangyong Huang Silong Yong Xiaojian Ma Xiongkun Linghu Puhao Li Yan Wang Qing Li Song-Chun Zhu Baoxiong Jia Siyuan Huang LM&Ro 23 131 0 18 Nov 2023
Offline Data Enhanced On-Policy Policy Gradient with Provable Guarantees Yifei Zhou Ayush Sekhari Yuda Song Wen Sun OffRL OnRL 25 8 0 14 Nov 2023
JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models Zihao Wang Shaofei Cai Anji Liu Yonggang Jin Jinbing Hou ... Zhaofeng He Zilong Zheng Yaodong Yang Xiaojian Ma Yitao Liang LLMAG LM&Ro 19 96 0 10 Nov 2023
ADaPT: As-Needed Decomposition and Planning with Language Models Archiki Prasad Alexander Koller Mareike Hartmann Peter Clark Ashish Sabharwal Mohit Bansal Tushar Khot LM&Ro 21 74 0 08 Nov 2023
Active Reasoning in an Open-World Environment Manjie Xu Guangyuan Jiang Weihan Liang Chi Zhang Yixin Zhu LLMAG LRM 8 10 0 03 Nov 2023
A Definition of Open-Ended Learning Problems for Goal-Conditioned Agents Olivier Sigaud Gianluca Baldassarre Cédric Colas Stéphane Doncieux Richard J. Duro Pierre-Yves Oudeyer Nicolas Perrin-Gilbert V. Santucci AI4CE 11 10 0 01 Nov 2023
Graph Agent: Explicit Reasoning Agent for Graphs Qinyong Wang Zhenxiang Gao Rong Xu AI4CE 21 5 0 25 Oct 2023
Steve-Eye: Equipping LLM-based Embodied Agents with Visual Perception in Open Worlds Sipeng Zheng Jiazheng Liu Yicheng Feng Zongqing Lu 31 29 0 20 Oct 2023
Eureka: Human-Level Reward Design via Coding Large Language Models Yecheng Jason Ma William Liang Guanzhi Wang De-An Huang Osbert Bastani Dinesh Jayaraman Yuke Zhu Linxi Fan A. Anandkumar 19 286 0 19 Oct 2023
Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning Juan Rocamonde Victoriano Montesinos Elvis Nava Ethan Perez David Lindner VLM 31 73 0 19 Oct 2023
AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agents Jake Grigsby Linxi Fan Yuke Zhu OffRL LM&Ro 24 10 0 15 Oct 2023
LLaMA Rider: Spurring Large Language Models to Explore the Open World Yicheng Feng Yuxuan Wang Jiazheng Liu Sipeng Zheng Zongqing Lu LLMAG LRM 16 15 0 13 Oct 2023
Octopus: Embodied Vision-Language Programmer from Environmental Feedback Jingkang Yang Yuhao Dong Shuai Liu Bo-wen Li Ziyue Wang ... Haoran Tan Jiamu Kang Yuanhan Zhang Kaiyang Zhou Ziwei Liu LM&Ro 36 45 0 12 Oct 2023
Cross-Episodic Curriculum for Transformer Agents Lucy Xiaoyang Shi Yunfan Jiang Jake Grigsby Linxi "Jim" Fan Yuke Zhu 10 4 0 12 Oct 2023
LightZero: A Unified Benchmark for Monte Carlo Tree Search in General Sequential Decision Scenarios Yazhe Niu Yuan Pu Zhenjie Yang Xueyan Li Tong Zhou Jiyuan Ren Shuai Hu Hongsheng Li Yu Liu 82 12 0 12 Oct 2023
GROOT: Learning to Follow Instructions by Watching Gameplay Videos Shaofei Cai Bowei Zhang Zihao Wang Xiaojian Ma Anji Liu Yitao Liang 83 26 0 12 Oct 2023
PonderV2: Pave the Way for 3D Foundation Model with A Universal Pre-training Paradigm Haoyi Zhu Honghui Yang Xiaoyang Wu Di Huang Sha Zhang ... Hengshuang Zhao Chunhua Shen Yu Qiao Tong He Wanli Ouyang SSL 69 42 0 12 Oct 2023
RoboCLIP: One Demonstration is Enough to Learn Robot Policies S. Sontakke Jesse Zhang Sébastien M. R. Arnold Karl Pertsch Erdem Biyik Dorsa Sadigh Chelsea Finn Laurent Itti OffRL 9 63 0 11 Oct 2023
Lemur: Harmonizing Natural Language and Code for Language Agents Yiheng Xu Hongjin Su Chen Xing Boyu Mi Qian Liu ... Siheng Zhao Lingpeng Kong Bailin Wang Caiming Xiong Tao Yu 27 67 0 10 Oct 2023
GEAR: A GPU-Centric Experience Replay System for Large Reinforcement Learning Models Hanjing Wang Man-Kit Sit Cong He Ying Wen Weinan Zhang J. Wang Yaodong Yang Luo Mai OffRL VLM 22 1 0 08 Oct 2023
Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models Xiaoxiao Sun Yang Yang Michal Shlapentokh-Rothman Haohan Wang Yu-xiong Wang LRM AI4CE LM&Ro LLMAG 29 183 0 06 Oct 2023
Towards End-to-End Embodied Decision Making via Multi-modal Large Language Model: Explorations with GPT4-Vision and Beyond Liang Chen Yichi Zhang Shuhuai Ren Haozhe Zhao Zefan Cai Yuchi Wang Peiyi Wang Tianyu Liu Baobao Chang LM&Ro LLMAG 16 41 0 03 Oct 2023
SmartPlay: A Benchmark for LLMs as Intelligent Agents Yue Wu Xuan Tang Tom Michael Mitchell Yuanzhi Li ELM LLMAG 27 63 0 02 Oct 2023
Berkeley Open Extended Reality Recordings 2023 (BOXRR-23): 4.7 Million Motion Capture Recordings from 105,852 Extended Reality Device Users V. Nair Wenbo Guo Rui Wang J. F. O'Brien Louis B. Rosenberg Dawn Song 13 7 0 30 Sep 2023
HarmonyDream: Task Harmonization Inside World Models Haoyu Ma Jialong Wu Ningya Feng Chenjun Xiao Dong Li Jianye Hao Jianmin Wang Mingsheng Long 33 7 0 30 Sep 2023
Motif: Intrinsic Motivation from Artificial Intelligence Feedback Martin Klissarov P. DÓro Shagun Sodhani Roberta Raileanu Pierre-Luc Bacon Pascal Vincent Amy Zhang Mikael Henaff LRM LLMAG 16 54 0 29 Sep 2023
Memory Gym: Towards Endless Tasks to Benchmark Memory Capabilities of Agents Marco Pleines Matthias Pallasch Frank Zimmer Mike Preuss OffRL 13 0 0 29 Sep 2023
Unsupervised 3D Perception with 2D Vision-Language Distillation for Autonomous Driving Mahyar Najibi Jingwei Ji Yin Zhou C. Qi Xinchen Yan Scott Ettinger Drago Anguelov 14 27 0 25 Sep 2023
An In-depth Survey of Large Language Model-based Artificial Intelligence Agents Pengyu Zhao Zijian Jin Ning Cheng LLMAG 30 19 0 23 Sep 2023
Text2Reward: Reward Shaping with Language Models for Reinforcement Learning Tianbao Xie Siheng Zhao Chen Henry Wu Yitao Liu Qian Luo Victor Zhong Yanchao Yang Tao Yu LM&Ro 31 48 0 20 Sep 2023
Guide Your Agent with Adaptive Multimodal Rewards Changyeon Kim Younggyo Seo Hao Liu Lisa Lee Jinwoo Shin Honglak Lee Kimin Lee 16 9 0 19 Sep 2023
A Data Source for Reasoning Embodied Agents Jack Lanchantin Sainbayar Sukhbaatar Gabriel Synnaeve Yuxuan Sun Kavya Srinet Arthur Szlam LM&Ro LRM 14 5 0 14 Sep 2023
Language Reward Modulation for Pretraining Reinforcement Learning Ademi Adeniji Amber Xie Carmelo Sferrazza Younggyo Seo Stephen James Pieter Abbeel 32 26 0 23 Aug 2023
ProAgent: Building Proactive Cooperative Agents with Large Language Models Ceyao Zhang Kaijie Yang Siyi Hu Zihao Wang Guanghe Li ... Xiaojun Chang Junge Zhang Feng Yin Yitao Liang Yaodong Yang LM&Ro LLMAG 28 69 0 22 Aug 2023
AgentBench: Evaluating LLMs as Agents Xiao Liu Hao Yu Hanchen Zhang Yifan Xu Xuanyu Lei ... Yu-Chuan Su Huan Sun Minlie Huang Yuxiao Dong Jie Tang ELM LLMAG 29 257 0 07 Aug 2023
Pre-Trained Large Language Models for Industrial Control Lei Song Chuheng Zhang Li Zhao Jiang Bian LM&Ro AI4CE 20 12 0 06 Aug 2023
Learning to Model the World with Language Jessy Lin Yuqing Du Olivia Watkins Danijar Hafner Pieter Abbeel Dan Klein Anca Dragan LM&Ro SyDa 13 48 0 31 Jul 2023
WebArena: A Realistic Web Environment for Building Autonomous Agents Shuyan Zhou Frank F. Xu Hao Zhu Xuhui Zhou Robert Lo ... Tianyue Ou Yonatan Bisk Daniel Fried Uri Alon Graham Neubig LLMAG 19 382 0 25 Jul 2023
Foundational Models Defining a New Era in Vision: A Survey and Outlook Muhammad Awais Muzammal Naseer Salman Khan Rao Muhammad Anwer Hisham Cholakkal M. Shah Ming Yang F. Khan VLM 18 116 0 25 Jul 2023
Towards A Unified Agent with Foundation Models Norman Di Palo Arunkumar Byravan Leonard Hasenclever Markus Wulfmeier N. Heess Martin Riedmiller LM&Ro LLMAG OffRL 30 58 0 18 Jul 2023
Is Imitation All You Need? Generalized Decision-Making with Dual-Phase Training Yao Wei Yanchao Sun Ruijie Zheng Sai H. Vemprala Rogerio Bonatti Shuhang Chen Ratnesh Madaan Zhongjie Ba Ashish Kapoor Shuang Ma OffRL 17 15 0 16 Jul 2023
RL-ViGen: A Reinforcement Learning Benchmark for Visual Generalization Zhecheng Yuan Sizhe Yang Pu Hua C. Chang Kaizhe Hu Huazhe Xu OOD OffRL 22 16 0 15 Jul 2023
Brain in a Vat: On Missing Pieces Towards Artificial General Intelligence in Large Language Models Yuxi Ma Chi Zhang Song-Chun Zhu ELM ALM 27 8 0 07 Jul 2023
Goal Representations for Instruction Following: A Semi-Supervised Language Interface to Control Vivek Myers Andre Wang He Kuan Fang Homer Walke Philippe Hansen-Estruch Ching-An Cheng Mihai Jalobeanu Andrey Kolobov Anca Dragan Sergey Levine LM&Ro 17 29 0 30 Jun 2023
Large Sequence Models for Sequential Decision-Making: A Survey Muning Wen Runji Lin Hanjing Wang Yaodong Yang Ying Wen Luo Mai J. Wang Haifeng Zhang Weinan Zhang LM&Ro LRM 29 34 0 24 Jun 2023