MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge

17 June 2022

Linxi Fan

De-An Huang

Papers citing "MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge"

50 / 286 papers shown

Title
EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agents Abhaysinh Zala Jaemin Cho Han Lin Jaehong Yoon Mohit Bansal 26 13 0 18 Mar 2024
A Survey on Game Playing Agents and Large Models: Methods, Applications, and Challenges Xinrun Xu Yuxin Wang Chaoyi Xu Ziluo Ding Jiechuan Jiang Zhiming Ding Börje F. Karlsson LM&Ro LLMAG 70 13 0 15 Mar 2024
USimAgent: Large Language Models for Simulating Search Users Erhan Zhang Xingzhu Wang Peiyuan Gong Yankai Lin Jiaxin Mao LLMAG 35 14 0 14 Mar 2024
Scaling Instructable Agents Across Many Simulated Worlds Sima Team Maria Abi Raad Arun Ahuja Catarina Barros F. Besse ... Daan Wierstra Duncan Williams Nathaniel Wong Sarah York Nick Young LM&Ro 107 35 0 13 Mar 2024
Hierarchical Auto-Organizing System for Open-Ended Multi-Agent Navigation Zhonghan Zhao Kewei Chen Dongxu Guo Wenhao Chai Tianbo Ye Yanting Zhang Gaoang Wang 53 20 0 13 Mar 2024
Online Continual Learning For Interactive Instruction Following Agents Byeonghwi Kim Minhyuk Seo Jonghyun Choi CLL LM&Ro 58 7 0 12 Mar 2024
OPEx: A Component-Wise Analysis of LLM-Centric Agents in Embodied Instruction Following Haochen Shi Zhiyuan Sun Xingdi Yuan Marc-Alexandre Côté Bang Liu LLMAG 21 10 0 05 Mar 2024
In-Memory Learning: A Declarative Learning Framework for Large Language Models Bo Wang Tianxiang Sun Hang Yan Siyin Wang Qingyuan Cheng Xipeng Qiu LLMAG 27 1 0 05 Mar 2024
AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attacks Jiacen Xu Jack W. Stokes Geoff McDonald Xuesong Bai David Marshall Siyue Wang Adith Swaminathan Zhou Li 35 48 0 02 Mar 2024
RL-GPT: Integrating Reinforcement Learning and Code-as-policy Shaoteng Liu Haoqi Yuan Minda Hu Yanwei Li Yukang Chen Shu Liu Zongqing Lu Jiaya Jia LLMAG 40 14 0 29 Feb 2024
On the Challenges and Opportunities in Generative AI Laura Manduchi Kushagra Pandey Robert Bamler Ryan Cotterell Sina Daubener ... F. Wenzel Frank Wood Stephan Mandt Vincent Fortuin Vincent Fortuin 56 17 0 28 Feb 2024
Empowering Large Language Model Agents through Action Learning Haiteng Zhao Chang Ma Guoyin Wang Jing Su Lingpeng Kong Jingjing Xu Zhi-Hong Deng Hongxia Yang LM&Ro LLMAG 49 9 0 24 Feb 2024
MORE-3S:Multimodal-based Offline Reinforcement Learning with Shared Semantic Spaces Tianyu Zheng Ge Zhang Xingwei Qu Ming Kuang Stephen W. Huang Zhaofeng He OffRL 29 1 0 20 Feb 2024
GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluations Jinhao Duan Renming Zhang James Diffenderfer B. Kailkhura Lichao Sun Elias Stengel-Eskin Mohit Bansal Tianlong Chen Kaidi Xu ELM LRM 32 55 0 19 Feb 2024
AQA-Bench: An Interactive Benchmark for Evaluating LLMs' Sequential Reasoning Ability Siwei Yang Bingchen Zhao Cihang Xie LRM 6 6 0 14 Feb 2024
AgentLens: Visual Analysis for Agent Behaviors in LLM-based Autonomous Systems Jiaying Lu Bo Pan Jieyi Chen Yingchaojie Feng Jingyuan Hu Yuchen Peng Wei Chen 34 13 0 14 Feb 2024
Towards Unified Alignment Between Agents, Humans, and Environment Zonghan Yang An Liu Zijun Liu Kai Liu Fangzhou Xiong ... Zhenhe Zhang Fuwen Luo Zhicheng Guo Peng Li Yang Liu 24 4 0 12 Feb 2024
OS-Copilot: Towards Generalist Computer Agents with Self-Improvement Zhiyong Wu Chengcheng Han Zichen Ding Zhenmin Weng Zhoumianze Liu Shunyu Yao Tao Yu Lingpeng Kong LLMAG LM&Ro 113 83 0 12 Feb 2024
Code as Reward: Empowering Reinforcement Learning with VLMs David Venuto Sami Nur Islam Martin Klissarov Doina Precup Sherry Yang Ankit Anand VLM 18 9 0 07 Feb 2024
S-Agents: Self-organizing Agents in Open-ended Environments Jia-Qing Chen Yu-Gang Jiang Jiachen Lu Li Zhang AIFin LLMAG LM&Ro 45 15 0 07 Feb 2024
Read to Play (R2-Play): Decision Transformer with Multimodal Game Instruction Yonggang Jin Ge Zhang Hao Zhao Tianyu Zheng Jiawei Guo Liuyu Xiang Shawn Yue Stephen W. Huang Zhaofeng He Jie Fu OffRL 24 4 0 06 Feb 2024
RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback Yufei Wang Zhanyi Sun Jesse Zhang Zhou Xian Erdem Biyik David Held Zackory M. Erickson VLM 53 48 0 06 Feb 2024
Assessing the Impact of Distribution Shift on Reinforcement Learning Performance Ted Fujimoto Joshua Suetterlein Samrat Chatterjee A. Ganguly OffRL 11 3 0 05 Feb 2024
Vision-Language Models Provide Promptable Representations for Reinforcement Learning William Chen Oier Mees Aviral Kumar Sergey Levine VLM LM&Ro 39 21 0 05 Feb 2024
Integration of cognitive tasks into artificial general intelligence test for large models Youzhi Qu Chen Wei Penghui Du Wenxin Che Chi Zhang ... Bin Hu Kai Du Haiyan Wu Jia Liu Quanying Liu ELM 34 6 0 04 Feb 2024
Point Cloud Matters: Rethinking the Impact of Different Observation Spaces on Robot Learning Haoyi Zhu Yating Wang Di Huang Weicai Ye Wanli Ouyang Tong He SSL 3DPC 31 18 0 04 Feb 2024
Computational Experiments Meet Large Language Model Based Agents: A Survey and Perspective Qun Ma Xiao Xue Deyu Zhou Xiangning Yu Donghua Liu ... Yifan Shen Peilin Ji Juanjuan Li Gang Wang Wanpeng Ma AI4CE LM&Ro LLMAG 11 6 0 01 Feb 2024
SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learning Jianlan Luo Zheyuan Hu Charles Xu You Liang Tan Jacob Berg Archit Sharma S. Schaal Chelsea Finn Abhishek Gupta Sergey Levine OffRL OnRL 34 40 0 29 Jan 2024
True Knowledge Comes from Practice: Aligning LLMs with Embodied Environments via Reinforcement Learning Weihao Tan Wentao Zhang Shanqi Liu Longtao Zheng Xinrun Wang Bo An OffRL 31 16 0 25 Jan 2024
SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities Boyuan Chen Zhuo Xu Sean Kirmani Brian Ichter Danny Driess Pete Florence Dorsa Sadigh Leonidas J. Guibas Fei Xia LRM ReLM 36 194 0 22 Jan 2024
CivRealm: A Learning and Reasoning Odyssey in Civilization for Decision-Making Agents Siyuan Qi Shuo Chen Yexin Li Xiangyu Kong Junqi Wang ... Zhaowei Zhang Nian Liu Wei Wang Yaodong Yang Song-Chun Zhu AI4CE LRM 19 17 0 19 Jan 2024
Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoning Yiqi Wang Wentao Chen Xiaotian Han Xudong Lin Haiteng Zhao Yongfei Liu Bohan Zhai Jianbo Yuan Quanzeng You Hongxia Yang LRM 33 66 0 10 Jan 2024
Language-Conditioned Robotic Manipulation with Fast and Slow Thinking Minjie Zhu Yichen Zhu Jinming Li Junjie Wen Zhiyuan Xu ... Chaomin Shen Yaxin Peng Dong Liu Feifei Feng Jian Tang LM&Ro 27 15 0 08 Jan 2024
Object-Centric Instruction Augmentation for Robotic Manipulation Junjie Wen Yichen Zhu Minjie Zhu Jinming Li Zhiyuan Xu ... Chaomin Shen Yaxin Peng Dong Liu Feifei Feng Jian Tang LM&Ro 62 16 0 05 Jan 2024
QUAR-VLA: Vision-Language-Action Model for Quadruped Robots Pengxiang Ding Han Zhao Wenxuan Song Zhitao Wang Zhenyu Wei Shangke Lyu Ningxi Yang Donglin Wang 25 19 0 22 Dec 2023
MinePlanner: A Benchmark for Long-Horizon Planning in Large Minecraft Worlds William Hill Ireton Liu Anita De Mello Koch Damion Harvey Nishanth Kumar G. Konidaris Steven D. James LM&Ro 9 0 0 20 Dec 2023
Auto MC-Reward: Automated Dense Reward Design with Large Language Models for Minecraft Hao Li Xue Yang Zhaokai Wang Xizhou Zhu Jie Zhou Yu Qiao Xiaogang Wang Hongsheng Li Lewei Lu Jifeng Dai 27 31 0 14 Dec 2023
Vision-Language Models as a Source of Rewards Kate Baumli Satinder Baveja Feryal M. P. Behbahani Harris Chan Gheorghe Comanici ... Yannick Schroecker Stephen Spencer Richie Steigerwald Luyu Wang Lei Zhang VLM LRM 32 26 0 14 Dec 2023
LiFT: Unsupervised Reinforcement Learning with Foundation Models as Teachers Taewook Nam Juyong Lee Jesse Zhang Sung Ju Hwang Joseph J. Lim Karl Pertsch OffRL LRM 43 4 0 14 Dec 2023
Foundation Models in Robotics: Applications, Challenges, and the Future Roya Firoozi Johnathan Tucker Stephen Tian Anirudha Majumdar Jiankai Sun ... Brian Ichter Danny Driess Jiajun Wu Cewu Lu Mac Schwager LM&Ro AI4CE LRM VLM 35 136 0 13 Dec 2023
MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perception Yiran Qin Enshen Zhou Qichang Liu Zhen-fei Yin Lu Sheng Ruimao Zhang Yu Qiao Jing Shao LM&Ro 17 39 0 12 Dec 2023
DiffVL: Scaling Up Soft Body Manipulation using Vision-Language Driven Differentiable Physics Zhiao Huang Feng Chen Yewen Pu Chun-Tse Lin Hao Su Chuang Gan 11 4 0 11 Dec 2023
Toward Open-ended Embodied Tasks Solving William Wei Wang Dongqi Han Xufang Luo Yifei Shen Charles X. Ling Boyu Wang Dongsheng Li AI4CE 8 5 0 10 Dec 2023
The Generalization Gap in Offline Reinforcement Learning Ishita Mediratta Qingfei You Minqi Jiang Roberta Raileanu OffRL 57 10 0 10 Dec 2023
DARLEI: Deep Accelerated Reinforcement Learning with Evolutionary Intelligence Saeejith Nair M. Shafiee Alexander Wong 8 0 0 08 Dec 2023
FoMo Rewards: Can we cast foundation models as reward functions? Ekdeep Singh Lubana Johann Brehmer P. D. Haan Taco S. Cohen OffRL LRM 33 2 0 06 Dec 2023
LLM as OS, Agents as Apps: Envisioning AIOS, Agents and the AIOS-Agent Ecosystem Yingqiang Ge Yujie Ren Wenyue Hua Shuyuan Xu Juntao Tan Yongfeng Zhang LLMAG 12 27 0 06 Dec 2023
Creative Agents: Empowering Agents with Imagination for Creative Tasks Chi Zhang Penglin Cai Yuhui Fu Haoqi Yuan Zongqing Lu LM&Ro LLMAG 49 20 0 05 Dec 2023
BEDD: The MineRL BASALT Evaluation and Demonstrations Dataset for Training and Benchmarking Agents that Solve Fuzzy Tasks Stephanie Milani Anssi Kanervisto Karolis Ramanauskas Sander Schulhoff Brandon Houghton Rohin Shah 21 6 0 05 Dec 2023
Quality Diversity in the Amorphous Fortress (QD-AF): Evolving for Complexity in 0-Player Games Sam Earle M. Charity Dipika Rajesh Mayu Wilson Julian Togelius 8 1 0 04 Dec 2023