Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenes

17 August 2023

Zehan Wang

Haifeng Huang

Yang Zhao

Ziang Zhang

Zhou Zhao

ArXiv PDF HTML

Papers citing "Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenes"

50 / 51 papers shown

Title
SpatialPrompting: Keyframe-driven Zero-Shot Spatial Reasoning with Off-the-Shelf Multimodal Large Language Models Shun Taguchi Hideki Deguchi Takumi Hamazaki Hiroyuki Sakai ReLM LRM 40 0 0 08 May 2025
The Point, the Vision and the Text: Does Point Cloud Boost Spatial Reasoning of Large Language Models? Weichen Zhang Ruiying Peng Chen Gao Jianjie Fang Xin Zeng ... Z. Wang Jinqiang Cui Xin Wang Xinlei Chen Y. Li LRM 71 0 0 06 Apr 2025
Ross3D: Reconstructive Visual Instruction Tuning with 3D-Awareness Haochen Wang Yucheng Zhao Tiancai Wang Haoqiang Fan X. Zhang Zhaoxiang Zhang 59 0 0 02 Apr 2025
ExCap3D: Expressive 3D Scene Understanding via Object Captioning with Varying Detail Chandan Yeshwanth Dávid Rozenberszki Angela Dai 71 0 0 21 Mar 2025
Exploring 3D Activity Reasoning and Planning: From Implicit Human Intentions to Route-Aware Planning Xueying Jiang Wenhao Li Xiaoqin Zhang Ling Shao Shijian Lu LRM 40 0 0 17 Mar 2025
SplatTalk: 3D VQA with Gaussian Splatting Anh Thai Songyou Peng Kyle Genova Leonidas J. Guibas Thomas Funkhouser 3DGS 75 0 0 08 Mar 2025
Bayesian Fields: Task-driven Open-Set Semantic Gaussian Splatting Dominic Maggio Luca Carlone 52 0 0 07 Mar 2025
Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuning Hanxun Yu Wentong Li Song Wang J. Chen Jianke Zhu 3DV LRM 71 3 0 01 Mar 2025
3D-AffordanceLLM: Harnessing Large Language Models for Open-Vocabulary Affordance Detection in 3D Worlds Hengshuo Chu Xiang Deng Qi Lv Xiaoyang Chen Yinchuan Li Jianye Hao Liqiang Nie 64 2 0 27 Feb 2025
Robin3D: Improving 3D Large Language Model via Robust Instruction Tuning Weitai Kang Haifeng Huang Yuzhang Shang Mubarak Shah Yan Yan 46 7 0 21 Feb 2025
LLMER: Crafting Interactive Extended Reality Worlds with JSON Data Generated by Large Language Models Jiangong Chen Xiaoyi Wu Tian Lan Bin Li 61 1 0 04 Feb 2025
GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models Zhangyang Qi Zhixiong Zhang Ye Fang Jiaqi Wang Hengshuang Zhao 83 6 0 02 Jan 2025
GaussianProperty: Integrating Physical Properties to 3D Gaussians with LMMs Xinli Xu Wenhang Ge Dicong Qiu ZhiFei Chen Dongyu Yan ... Haoyu Zhao HanFeng Zhao Shunsi Zhang Junwei Liang Ying-cong Chen 3DGS 77 1 0 15 Dec 2024
LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferences Hongyan Zhi Peihao Chen Junyan Li Shuailei Ma Xinyu Sun Tianhang Xiang Yinjie Lei Mingkui Tan Chuang Gan 67 3 0 02 Dec 2024
SPARTUN3D: Situated Spatial Understanding of 3D World in Large Language Models Yue Zhang Zhiyang Xu Ying Shen Parisa Kordjamshidi Lifu Huang 32 6 0 04 Oct 2024
ELLMA-T: an Embodied LLM-agent for Supporting English Language Learning in Social VR Mengxu Pan Alexandra Kitson Hongyu Wan Mirjana Prpa 28 2 0 03 Oct 2024
Grounding 3D Scene Affordance From Egocentric Interactions Cuiyu Liu Wei Zhai Yuhang Yang Hongchen Luo Sen Liang Yang Cao Zheng-Jun Zha 26 1 0 29 Sep 2024
SYNERGAI: Perception Alignment for Human-Robot Collaboration Yixin Chen Guoxi Zhang Yaowei Zhang Hongming Xu Peiyuan Zhi Qing Li Siyuan Huang 32 0 0 24 Sep 2024
SAM2Point: Segment Any 3D as Videos in Zero-shot and Promptable Manners Ziyu Guo Renrui Zhang Xiangyang Zhu Chengzhuo Tong Peng Gao Chunyuan Li Pheng-Ann Heng VGen 3DPC 42 9 0 29 Aug 2024
LLM-enhanced Scene Graph Learning for Household Rearrangement Wenhao Li Zhiyuan Yu Qijin She Zhinan Yu Yuqing Lan Chenyang Zhu Ruizhen Hu Kai Xu 36 1 0 22 Aug 2024
OmniBind: Large-scale Omni Multimodal Representation via Binding Spaces Zehan Wang Ziang Zhang Hang Zhang Luping Liu Rongjie Huang Xize Cheng Hengshuang Zhao Zhou Zhao 27 7 0 16 Jul 2024
Affordance-Guided Reinforcement Learning via Visual Prompting Olivia Y. Lee Annie Xie Kuan Fang Karl Pertsch Chelsea Finn OffRL LM&Ro 62 7 0 14 Jul 2024
Slice-100K: A Multimodal Dataset for Extrusion-based 3D Printing Anushrut Jignasu Kelly O. Marshall Ankush Kumar Mishra Lucas Nerone Rillo Baskar Ganapathysubramanian Aditya Balu Chinmay Hegde Adarsh Krishnamurthy 24 0 0 04 Jul 2024
The Effects of Embodiment and Personality Expression on Learning in LLM-based Educational Agents Sinan Sonlu Bennie Bendiksen Funda Durupinar U. Güdükbay 26 7 0 24 Jun 2024
MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations Ruiyuan Lyu Tai Wang Jingli Lin Shuai Yang Xiaohan Mao ... Runsen Xu Haifeng Huang Chenming Zhu Dahua Lin Jiangmiao Pang 3DV 36 9 0 13 Jun 2024
VP-LLM: Text-Driven 3D Volume Completion with Large Language Models through Patchification Jianmeng Liu Yichen Liu Yuyao Zhang Zeyuan Meng Yu-Wing Tai Chi-Keung Tang 42 0 0 08 Jun 2024
3D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucination Jianing Yang Xuweiyi Chen Nikhil Madaan Madhavan Iyengar Shengyi Qian David Fouhey Joyce Chai 3DV 61 11 0 07 Jun 2024
RoboMamba: Multimodal State Space Model for Efficient Robot Reasoning and Manipulation Jiaming Liu Mengzhen Liu Zhenyu Wang Lily Lee Kaichen Zhou Pengju An Senqiao Yang Renrui Zhang Yandong Guo Shanghang Zhang LM&Ro LRM Mamba 27 5 0 06 Jun 2024
Reasoning3D -- Grounding and Reasoning in 3D: Fine-Grained Zero-Shot Open-Vocabulary 3D Reasoning Part Segmentation via Large Vision-Language Models Tianrun Chen Chunan Yu Jing Li Jianqi Zhang Lanyun Zhu Deyi Ji Yong Zhang Ying-Dong Zang Zejian Li Lingyun Sun LRM 36 9 0 29 May 2024
Intent3D: 3D Object Detection in RGB-D Scans Based on Human Intention Weitai Kang Mengxue Qu Jyoti Kini Yunchao Wei Mubarak Shah Yan Yan LM&Ro 3DPC 39 9 0 28 May 2024
LLM-Optic: Unveiling the Capabilities of Large Language Models for Universal Visual Grounding Haoyu Zhao Wenhang Ge Ying-cong Chen ObjD MLLM VLM 24 1 0 27 May 2024
Grounded 3D-LLM with Referent Tokens Yilun Chen Shuai Yang Haifeng Huang Tai Wang Ruiyuan Lyu Runsen Xu Dahua Lin Jiangmiao Pang 45 22 0 16 May 2024
FreeBind: Free Lunch in Unified Multimodal Space via Knowledge Fusion Zehan Wang Ziang Zhang Xize Cheng Rongjie Huang Luping Liu ... Haifeng Huang Yang Zhao Tao Jin Peng Gao Zhou Zhao 18 8 0 08 May 2024
PARIS3D: Reasoning-based 3D Part Segmentation Using Large Multimodal Model Amrin Kareem Jean Lahoud Hisham Cholakkal LRM 39 3 0 04 Apr 2024
Agent3D-Zero: An Agent for Zero-shot 3D Understanding Sha Zhang Di Huang Jiajun Deng Shixiang Tang Wanli Ouyang Tong He Yanyong Zhang VGen 28 13 0 18 Mar 2024
MM-LLMs: Recent Advances in MultiModal Large Language Models Duzhen Zhang Yahan Yu Jiahua Dong Chenxing Li Dan Su Chenhui Chu Dong Yu OffRL LRM 37 173 0 24 Jan 2024
3DMIT: 3D Multi-modal Instruction Tuning for Scene Understanding Zeju Li Chao Zhang Xiaoyan Wang Ruilong Ren Yifan Xu Ruifei Ma Xiangde Liu MLLM 13 20 0 06 Jan 2024
LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understanding Senqiao Yang Jiaming Liu Ray Zhang Mingjie Pan Zoey Guo Xiaoqi Li Zehui Chen Peng Gao Yandong Guo Shanghang Zhang 3DV 8 58 0 21 Dec 2023
M3DBench: Let's Instruct Large Models with Multi-modal 3D Prompts Mingsheng Li Xin Chen C. Zhang Sijin Chen Hongyuan Zhu Fukun Yin Gang Yu Tao Chen 17 23 0 17 Dec 2023
3DAxiesPrompts: Unleashing the 3D Spatial Task Capabilities of GPT-4V Dingning Liu Xiaomeng Dong Renrui Zhang Xu Luo Peng Gao Xiaoshui Huang Yongshun Gong Zhihui Wang 19 8 0 15 Dec 2023
Chat-3D v2: Bridging 3D Scene and Large Language Models with Object Identifiers Haifeng Huang Zehan Wang Rongjie Huang Luping Liu Xize Cheng Yang Zhao Tao Jin Zhou Zhao 50 40 0 13 Dec 2023
An Embodied Generalist Agent in 3D World Jiangyong Huang Silong Yong Xiaojian Ma Xiongkun Linghu Puhao Li Yan Wang Qing Li Song-Chun Zhu Baoxiong Jia Siyuan Huang LM&Ro 20 131 0 18 Nov 2023
Extending Multi-modal Contrastive Representations Zehan Wang Ziang Zhang Luping Liu Yang Zhao Haifeng Huang Tao Jin Zhou Zhao 19 5 0 13 Oct 2023
LLMR: Real-time Prompting of Interactive Worlds using Large Language Models Fernanda De La Torre Cathy Mengying Fang Han Huang Andrzej Banburski-Fahey Judith Amores Fernandez Jaron Lanier 21 45 0 21 Sep 2023
Connecting Multi-modal Contrastive Representations Zehan Wang Yang Zhao Xize Cheng Haifeng Huang Jiageng Liu ... Lin Li Yongqiang Wang Aoxiong Yin Ziang Zhang Zhou Zhao 14 11 0 22 May 2023
mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality Qinghao Ye Haiyang Xu Guohai Xu Jiabo Ye Ming Yan ... Junfeng Tian Qiang Qi Ji Zhang Feiyan Huang Jingren Zhou VLM MLLM 203 883 0 27 Apr 2023
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models Junnan Li Dongxu Li Silvio Savarese Steven C. H. Hoi VLM MLLM 244 4,186 0 30 Jan 2023
Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training Renrui Zhang Ziyu Guo Rongyao Fang Bingyan Zhao Dong Wang Yu Qiao Hongsheng Li Peng Gao 3DPC 167 241 0 28 May 2022
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation Junnan Li Dongxu Li Caiming Xiong S. Hoi MLLM BDL VLM CLIP 382 4,010 0 28 Jan 2022
Ego4D: Around the World in 3,000 Hours of Egocentric Video Kristen Grauman Andrew Westbury Eugene Byrne Zachary Chavis Antonino Furnari ... Mike Zheng Shou Antonio Torralba Lorenzo Torresani Mingfei Yan Jitendra Malik EgoV 218 682 0 13 Oct 2021