ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understanding

14 May 2023

Roberto Martín-Martín

Jiajun Wu

Caiming Xiong

Ran Xu

Juan Carlos Niebles

Silvio Savarese

ArXiv PDF HTML

Papers citing "ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understanding"

49 / 99 papers shown

Title
SUGAR: Pre-training 3D Visual Representations for Robotics Shizhe Chen Ricardo Garcia Pinel Ivan Laptev Cordelia Schmid 37 13 0 01 Apr 2024
PointSeg: A Training-Free Paradigm for 3D Scene Segmentation via Foundation Models Qingdong He Jinlong Peng Zhengkai Jiang Xiaobin Hu Jiangning Zhang Qiang Nie Yabiao Wang Chengjie Wang 3DPC VLM 41 5 0 11 Mar 2024
3DTopia: Large Text-to-3D Generation Model with Hybrid Diffusion Priors Fangzhou Hong Jiaxiang Tang Ziang Cao Min Shi Tong Wu ... Shuai Yang Tengfei Wang Liang Pan Dahua Lin Ziwei Liu 30 44 0 04 Mar 2024
ShapeLLM: Universal 3D Object Understanding for Embodied Interaction Zekun Qi Runpei Dong Shaochen Zhang Haoran Geng Chunrui Han Zheng Ge Li Yi Kaisheng Ma 39 49 0 27 Feb 2024
Parameter-efficient Prompt Learning for 3D Point Cloud Understanding Hongyu Sun Yongcai Wang Wang Chen Haoran Deng Deying Li VPVLM 39 5 0 24 Feb 2024
CLIPose: Category-Level Object Pose Estimation with Pre-trained Vision-Language Knowledge Xiao Lin Minghao Zhu Ronghao Dang Guangliang Zhou Shaolong Shu Feng Lin Chengju Liu Qi Chen CLIP 35 6 0 24 Feb 2024
GS-CLIP: Gaussian Splatting for Contrastive Language-Image-3D Pretraining from Real-World Data Haoyuan Li Yanpeng Zhou Yihan Zeng Hang Xu Xiaodan Liang 3DGS CLIP 13 0 0 09 Feb 2024
Transolver: A Fast Transformer Solver for PDEs on General Geometries Haixu Wu Huakun Luo Haowen Wang Jianmin Wang Mingsheng Long AI4CE 35 39 0 04 Feb 2024
UniM-OV3D: Uni-Modality Open-Vocabulary 3D Scene Understanding with Fine-Grained Feature Representation Qingdong He Jinlong Peng Zhengkai Jiang Kai Wu Xiaozhong Ji Jiangning Zhang Yabiao Wang Chengjie Wang Mingang Chen Yunsheng Wu 3DPC 13 7 0 21 Jan 2024
3DMIT: 3D Multi-modal Instruction Tuning for Scene Understanding Zeju Li Chao Zhang Xiaoyan Wang Ruilong Ren Yifan Xu Ruifei Ma Xiangde Liu MLLM 13 20 0 06 Jan 2024
From Google Gemini to OpenAI Q* (Q-Star): A Survey of Reshaping the Generative Artificial Intelligence (AI) Research Landscape Timothy R. McIntosh Teo Susnjak Tong Liu Paul Watters Malka N. Halgamuge 79 46 0 18 Dec 2023
Pedestrian Attribute Recognition via CLIP based Prompt Vision-Language Fusion Xiao Wang Jiandong Jin Chenglong Li Jin Tang Cheng Zhang Wei Wang VLM 15 13 0 17 Dec 2023
3DAxiesPrompts: Unleashing the 3D Spatial Task Capabilities of GPT-4V Dingning Liu Xiaomeng Dong Renrui Zhang Xu Luo Peng Gao Xiaoshui Huang Yongshun Gong Zhihui Wang 27 10 0 15 Dec 2023
Foundation Models in Robotics: Applications, Challenges, and the Future Roya Firoozi Johnathan Tucker Stephen Tian Anirudha Majumdar Jiankai Sun ... Brian Ichter Danny Driess Jiajun Wu Cewu Lu Mac Schwager LM&Ro AI4CE LRM VLM 35 136 0 13 Dec 2023
GPT4Point: A Unified Framework for Point-Language Understanding and Generation Zhangyang Qi Ye Fang Zeyi Sun Xiaoyang Wu Tong Wu Jiaqi Wang Dahua Lin Hengshuang Zhao MLLM 71 35 0 05 Dec 2023
Uni3DL: Unified Model for 3D and Language Understanding Xiang Li Jian Ding Zhaoyang Chen Mohamed Elhoseiny 26 3 0 05 Dec 2023
Language Embedded 3D Gaussians for Open-Vocabulary Scene Understanding Jin-Chuan Shi Miao Wang Hao-Bin Duan Shao-Hua Guan 3DGS 25 83 0 30 Nov 2023
MV-CLIP: Multi-View CLIP for Zero-shot 3D Shape Recognition Dan Song Xinwei Fu Weizhi Nie Wenhui Li Lanjun Wang You Yang Anan Liu VLM 22 6 0 30 Nov 2023
ViT-Lens: Towards Omni-modal Representations Weixian Lei Yixiao Ge Kun Yi Jianfeng Zhang Difei Gao Dylan Sun Yuying Ge Ying Shan Mike Zheng Shou 21 18 0 27 Nov 2023
Applications of Large Scale Foundation Models for Autonomous Driving Yu Huang Yue Chen Zhu Li ELM AI4CE LRM ALM LM&Ro 46 14 0 20 Nov 2023
OmniVec: Learning robust representations with cross modal sharing Siddharth Srivastava Gaurav Sharma SSL 16 64 0 07 Nov 2023
Sculpting Holistic 3D Representation in Contrastive Language-Image-3D Pre-training Yipeng Gao Zeyu Wang Wei-Shi Zheng Cihang Xie Yuyin Zhou 3DPC 19 8 0 03 Nov 2023
RoomDesigner: Encoding Anchor-latents for Style-consistent and Shape-compatible Indoor Scene Generation Yiqun Zhao Zibo Zhao Jing Li Sixun Dong Shenghua Gao 3DV 25 9 0 16 Oct 2023
JM3D & JM3D-LLM: Elevating 3D Understanding with Joint Multi-modal Cues Jiayi Ji Haowei Wang Changli Wu Yiwei Ma Xiaoshuai Sun Rongrong Ji 32 1 0 14 Oct 2023
Extending Multi-modal Contrastive Representations Zehan Wang Ziang Zhang Luping Liu Yang Zhao Haifeng Huang Tao Jin Zhou Zhao 19 5 0 13 Oct 2023
Uni3D: Exploring Unified 3D Representation at Scale Junsheng Zhou Jinsheng Wang Baorui Ma Yu-Shen Liu Tiejun Huang Xinlong Wang 32 86 0 10 Oct 2023
Text-to-3D using Gaussian Splatting Manish Sharma Moitreya Chatterjee Yikai Wang Huaping Liu 3DGS 20 220 0 28 Sep 2023
PolarNet: 3D Point Clouds for Language-Guided Robotic Manipulation Shizhe Chen Ricardo Garcia Pinel Cordelia Schmid Ivan Laptev LM&Ro 3DPC 25 33 0 27 Sep 2023
Looking at words and points with attention: a benchmark for text-to-shape coherence Andrea Amaduzzi Giuseppe Lisanti Samuele Salti Luigi Di Stefano 11 2 0 14 Sep 2023
Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following Ziyu Guo Renrui Zhang Xiangyang Zhu Yiwen Tang Xianzheng Ma ... Ke Chen Peng Gao Xianzhi Li Hongsheng Li Pheng-Ann Heng MLLM 17 123 0 01 Sep 2023
PointLLM: Empowering Large Language Models to Understand Point Clouds Runsen Xu Xiaolong Wang Tai Wang Yilun Chen Jiangmiao Pang Dahua Lin MLLM 51 146 0 31 Aug 2023
ViT-Lens: Initiating Omni-Modal Exploration through 3D Insights Weixian Lei Yixiao Ge Jianfeng Zhang Dylan Sun Kun Yi Ying Shan Mike Zheng Shou 25 1 0 20 Aug 2023
Fine-grained Text and Image Guided Point Cloud Completion with CLIP Model Wei Song Jun Zhou Mingjie Wang Hongchen Tan Nannan Li Xiuping Liu 10 2 0 17 Aug 2023
Human Motion Generation: A Survey Wentao Zhu Xiaoxuan Ma Dongwoo Ro Hai Ci Jinlu Zhang Jiaxin Shi Feng Gao Qi Tian Yizhou Wang VGen 26 50 0 20 Jul 2023
Objaverse-XL: A Universe of 10M+ 3D Objects Matt Deitke Ruoshi Liu Matthew Wallingford Huong Ngo Oscar Michel ... Carl Vondrick Georgia Gkioxari Kiana Ehsani Ludwig Schmidt Ali Farhadi 18 377 0 11 Jul 2023
Multi-task 3D building understanding with multi-modal pretraining Shicheng Xu 3DPC 6 2 0 16 Jun 2023
Scalable 3D Captioning with Pretrained Models Tiange Luo C. Rockwell Honglak Lee Justin Johnson 16 151 0 12 Jun 2023
A Survey of Label-Efficient Deep Learning for 3D Point Clouds Aoran Xiao Xiaoqin Zhang Ling Shao Shijian Lu 3DPC 27 18 0 31 May 2023
Connecting Multi-modal Contrastive Representations Zehan Wang Yang Zhao Xize Cheng Haifeng Huang Jiageng Liu ... Lin Li Yongqiang Wang Aoxiong Yin Ziang Zhang Zhou Zhao 17 22 0 22 May 2023
Let Images Give You More:Point Cloud Cross-Modal Training for Shape Analysis Xu Yan Heshen Zhan Chaoda Zheng Jiantao Gao Ruimao Zhang Shuguang Cui Zhen Li 3DPC 49 32 0 09 Oct 2022
Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training Renrui Zhang Ziyu Guo Rongyao Fang Bingyan Zhao Dong Wang Yu Qiao Hongsheng Li Peng Gao 3DPC 171 241 0 28 May 2022
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation Junnan Li Dongxu Li Caiming Xiong S. Hoi MLLM BDL VLM CLIP 388 4,010 0 28 Jan 2022
PointCLIP: Point Cloud Understanding by CLIP Renrui Zhang Ziyu Guo Wei Zhang Kunchang Li Xupeng Miao Bin Cui Yu Qiao Peng Gao Hongsheng Li VLM 3DPC 164 428 0 04 Dec 2021
Unifying Vision-and-Language Tasks via Text Generation Jaemin Cho Jie Lei Hao Tan Mohit Bansal MLLM 249 525 0 04 Feb 2021
Unified Vision-Language Pre-Training for Image Captioning and VQA Luowei Zhou Hamid Palangi Lei Zhang Houdong Hu Jason J. Corso Jianfeng Gao MLLM VLM 250 922 0 24 Sep 2019
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 243 1,791 0 17 Sep 2019
DensePoint: Learning Densely Contextual Representation for Efficient Point Cloud Processing Yongcheng Liu Bin Fan Gaofeng Meng Jiwen Lu Shiming Xiang Chunhong Pan 3DPC 115 269 0 09 Sep 2019
Neural Baby Talk Jiasen Lu Jianwei Yang Dhruv Batra Devi Parikh VLM 189 432 0 27 Mar 2018
PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation C. Qi Hao Su Kaichun Mo Leonidas J. Guibas 3DH 3DPC 3DV PINN 219 13,886 0 02 Dec 2016