VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models

12 July 2023

Wenlong Huang

Chen Wang

Ruohan Zhang

Yunzhu Li

Jiajun Wu

Li Fei-Fei

LM&Ro

ArXiv PDF HTML

Papers citing "VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models"

50 / 108 papers shown

Title
Affordance-Guided Reinforcement Learning via Visual Prompting Olivia Y. Lee Annie Xie Kuan Fang Karl Pertsch Chelsea Finn OffRL LM&Ro 64 7 0 14 Jul 2024
VLMPC: Vision-Language Model Predictive Control for Robotic Manipulation Wentao Zhao Jiaming Chen Ziyu Meng Donghui Mao Ran Song Wei Zhang 35 8 0 13 Jul 2024
LLaRA: Supercharging Robot Learning Data for Vision-Language Policy Xiang Li Cristina Mata J. Park Kumara Kahatapitiya Yoo Sung Jang ... Kanchana Ranasinghe R. Burgert Mu Cai Yong Jae Lee Michael S. Ryoo LM&Ro 59 25 0 28 Jun 2024
LiveScene: Language Embedding Interactive Radiance Fields for Physical Scene Rendering and Control Delin Qu Qizhi Chen Pingrui Zhang Xianqiang Gao Bin Zhao Bin Zhao Dong Wang Xuelong Li AI4CE 34 7 0 23 Jun 2024
FreeMotion: MoCap-Free Human Motion Synthesis with Multimodal Large Language Models Zhikai Zhang Yitang Li Haofeng Huang Mingxian Lin Li Yi 35 2 0 15 Jun 2024
3D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucination Jianing Yang Xuweiyi Chen Nikhil Madaan Madhavan Iyengar Shengyi Qian David Fouhey Joyce Chai 3DV 68 11 0 07 Jun 2024
Evaluating Uncertainty-based Failure Detection for Closed-Loop LLM Planners Zhi Zheng Qian Feng Hang Li Alois C. Knoll Jianxiang Feng 44 6 0 01 Jun 2024
Robo-Instruct: Simulator-Augmented Instruction Alignment For Finetuning Code LLMs Zichao Hu Junyi Jessy Li Arjun Guha Joydeep Biswas SyDa ALM 37 1 0 30 May 2024
Towards Efficient LLM Grounding for Embodied Multi-Agent Collaboration Yang Zhang Shixin Yang Chenjia Bai Fei Wu Xiu Li Zhen Wang Xuelong Li LLMAG 31 25 0 23 May 2024
A Survey on Vision-Language-Action Models for Embodied AI Yueen Ma Zixing Song Yuzheng Zhuang Jianye Hao Irwin King LM&Ro 64 41 0 23 May 2024
Innovative Integration of Visual Foundation Model with a Robotic Arm on a Mobile Platform Shimian Zhang Qiuhong Lu 21 1 0 29 Apr 2024
Agile and versatile bipedal robot tracking control through reinforcement learning Jiayi Li Linqi Ye Yi Cheng Houde Liu Bin Liang 23 1 0 12 Apr 2024
Chat Modeling: Natural Language-based Procedural Modeling of Biological Structures without Training Donggang Jia Yunhai Wang Ivan Viola 26 1 0 01 Apr 2024
Context-aware LLM-based Safe Control Against Latent Risks Quang Khanh Luu Xiyu Deng Anh Van Ho Yorie Nakahira 51 4 0 18 Mar 2024
ExploRLLM: Guiding Exploration in Reinforcement Learning with Large Language Models Runyu Ma Jelle Luijkx Zlatan Ajanović Jens Kober LM&Ro LRM 33 7 0 14 Mar 2024
Proxy-RLHF: Decoupling Generation and Alignment in Large Language Model with Proxy Yu Zhu Chuxiong Sun Wenfei Yang Wenqiang Wei Bo Tang ... Zhiyu Li Shifeng Zhang Feiyu Xiong Jie Hu Mingchuan Yang 29 3 0 07 Mar 2024
CyberDemo: Augmenting Simulated Human Demonstration for Real-World Dexterous Manipulation Jun Wang Yuzhe Qin Kaiming Kuang Yigit Korkmaz Akhilan Gurumoorthy Hao Su Xiaolong Wang 36 19 0 22 Feb 2024
RoboScript: Code Generation for Free-Form Manipulation Tasks across Real and Simulation Junting Chen Yao Mu Qiaojun Yu Tianming Wei Silang Wu ... Wenqi Shao Yu Qiao Huazhe Xu Mingyu Ding Ping Luo LM&Ro 25 11 0 22 Feb 2024
Verifiably Following Complex Robot Instructions with Foundation Models Benedict Quartey Eric Rosen Stefanie Tellex G. Konidaris LM&Ro 39 10 0 18 Feb 2024
Towards Unified Alignment Between Agents, Humans, and Environment Zonghan Yang An Liu Zijun Liu Kai Liu Fangzhou Xiong ... Zhenhe Zhang Fuwen Luo Zhicheng Guo Peng Li Yang Liu 24 4 0 12 Feb 2024
ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation Xiaoqi Li Mingxu Zhang Yiran Geng Haoran Geng Yuxing Long Yan Shen Renrui Zhang Jiaming Liu Hao Dong LM&Ro LRM 25 78 0 24 Dec 2023
Building Open-Ended Embodied Agent via Language-Policy Bidirectional Adaptation Shaopeng Zhai Jie Wang Tianyi Zhang Fuxian Huang Qi Zhang Ming Zhou Jing Hou Yu Qiao Yu Liu LLMAG LM&Ro 19 1 0 12 Dec 2023
ADriver-I: A General World Model for Autonomous Driving Fan Jia Weixin Mao Yingfei Liu Yucheng Zhao Yuqing Wen Chi Zhang Xiangyu Zhang Tiancai Wang 22 63 0 22 Nov 2023
Teaching Robots to Build Simulations of Themselves Yuhang Hu Jiong Lin Hod Lipson SSL 19 4 0 20 Nov 2023
Distilling and Retrieving Generalizable Knowledge for Robot Manipulation via Language Corrections Lihan Zha Yuchen Cui Li-Heng Lin Minae Kwon Montse Gonzalez Arenas Andy Zeng Fei Xia Dorsa Sadigh 23 34 0 17 Nov 2023
Advances in Embodied Navigation Using Large Language Models: A Survey Jinzhou Lin Han Gao Xuxiang Feng Rongtao Xu Changwei Wang Man Zhang Li Guo Shibiao Xu LM&Ro LLMAG 56 9 0 01 Nov 2023
Learning Reward for Physical Skills using Large Language Model Yuwei Zeng Yiqing Xu 23 6 0 21 Oct 2023
Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning Juan Rocamonde Victoriano Montesinos Elvis Nava Ethan Perez David Lindner VLM 31 73 0 19 Oct 2023
LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving Hao Sha Yao Mu Yuxuan Jiang Li Chen Chenfeng Xu Ping Luo Shengbo Eben Li Masayoshi Tomizuka Wei Zhan Mingyu Ding 107 159 0 04 Oct 2023
QwenGrasp: A Usage of Large Vision-Language Model for Target-Oriented Grasping Xinyu Chen Jian Yang Zonghan He Sascha Rank Qi Zhao Yuhui Shi 24 3 0 28 Sep 2023
OceanChat: Piloting Autonomous Underwater Vehicles in Natural Language Jia Huang Mengxue Hou Junkai Wang Fumin Zhang 16 5 0 27 Sep 2023
HiCRISP: An LLM-based Hierarchical Closed-Loop Robotic Intelligent Self-Correction Planner Chenlin Ming Jiacheng Lin Pangkit Fong Han Wang Xiaoming Duan Jianping He 15 1 0 21 Sep 2023
Are Large Language Models Really Robust to Word-Level Perturbations? Haoyu Wang Guozheng Ma Cong Yu Ning Gui Linrui Zhang ... Sen Zhang Li Shen Xueqian Wang Peilin Zhao Dacheng Tao KELM 21 22 0 20 Sep 2023
LARG, Language-based Automatic Reward and Goal Generation Julien Perez Denys Proux Claude Roux Michael Niemaz LM&Ro 12 1 0 19 Jun 2023
Transferring Foundation Models for Generalizable Robotic Manipulation Jiange Yang Wenhui Tan Chuhao Jin Keling Yao Bei Liu Jianlong Fu Ruihua Song Gangshan Wu Limin Wang LM&Ro 45 6 0 09 Jun 2023
Multimodal Procedural Planning via Dual Text-Image Prompting Yujie Lu Pan Lu Zhiyu Zoey Chen Wanrong Zhu X. Wang William Yang Wang LM&Ro 59 43 0 02 May 2023
Prophet: Prompting Large Language Models with Complementary Answer Heuristics for Knowledge-based Visual Question Answering Zhou Yu Xuecheng Ouyang Zhenwei Shao Mei Wang Jun Yu MLLM 86 11 0 03 Mar 2023
Open-World Object Manipulation using Pre-trained Vision-Language Models Austin Stone Ted Xiao Yao Lu K. Gopalakrishnan Kuang-Huei Lee ... Sean Kirmani Brianna Zitkovich F. Xia Chelsea Finn Karol Hausman LM&Ro 142 144 0 02 Mar 2023
Visual Language Maps for Robot Navigation Chen Huang Oier Mees Andy Zeng Wolfram Burgard LM&Ro 145 337 0 11 Oct 2022
Grounding Language with Visual Affordances over Unstructured Data Oier Mees Jessica Borja-Diaz Wolfram Burgard LM&Ro 121 106 0 04 Oct 2022
ProgPrompt: Generating Situated Robot Task Plans using Large Language Models Ishika Singh Valts Blukis Arsalan Mousavian Ankit Goyal Danfei Xu Jonathan Tremblay D. Fox Jesse Thomason Animesh Garg LM&Ro LLMAG 112 616 0 22 Sep 2022
Open-vocabulary Queryable Scene Representations for Real World Planning Boyuan Chen F. Xia Brian Ichter Kanishka Rao K. Gopalakrishnan Michael S. Ryoo Austin Stone Daniel Kappler LM&Ro 144 179 0 20 Sep 2022
Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation Mohit Shridhar Lucas Manuelli D. Fox LM&Ro 146 449 0 12 Sep 2022
LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action Dhruv Shah B. Osinski Brian Ichter Sergey Levine LM&Ro 139 430 0 10 Jul 2022
Large Language Models are Zero-Shot Reasoners Takeshi Kojima S. Gu Machel Reid Yutaka Matsuo Yusuke Iwasawa ReLM LRM 291 4,048 0 24 May 2022
Generalizable Task Planning through Representation Pretraining Chen Wang Danfei Xu Li Fei-Fei LM&Ro 53 27 0 16 May 2022
Learning to Reorient Objects with Stable Placements Afforded by Extrinsic Supports Peng Xu Hu Cheng Jiankun Wang M. Meng 30 4 0 14 May 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 303 11,730 0 04 Mar 2022
Improving Intrinsic Exploration with Language Abstractions Jesse Mu Victor Zhong Roberta Raileanu Minqi Jiang Noah D. Goodman Tim Rocktaschel Edward Grefenstette 95 63 0 17 Feb 2022
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason W. Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter F. Xia Ed H. Chi Quoc Le Denny Zhou LM&Ro LRM AI4CE ReLM 315 8,261 0 28 Jan 2022