Towards Semantic Equivalence of Tokenization in Multimodal LLM

7 June 2024

Xiangtai Li

Hanwang Zhang

Papers citing "Towards Semantic Equivalence of Tokenization in Multimodal LLM"

24 / 24 papers shown

Title
Position: Foundation Models Need Digital Twin Representations Yiqing Shen Hao Ding Lalithkumar Seenivasan Tianmin Shu Mathias Unberath AI4CE 27 0 0 01 May 2025
AlphaSpace: Enabling Robotic Actions through Semantic Tokenization and Symbolic Reasoning Alan Dao Dinh Bach Vu Bui Quang Huy 48 0 0 24 Mar 2025
Task-Oriented Feature Compression for Multimodal Understanding via Device-Edge Co-Inference Cheng Yuan Z. Liu Jiashu Lv Jiawei Shao Yufei Jiang J. Zhang Xuelong Li 38 0 0 17 Mar 2025
Multi-Granular Multimodal Clue Fusion for Meme Understanding Li Zheng Hao Fei Ting Dai Zuquan Peng Fei Li Huisheng Ma Chong Teng Donghong Ji 48 0 0 16 Mar 2025
CalliReader: Contextualizing Chinese Calligraphy via an Embedding-Aligned Vision-Language Model Yuxuan Luo Jiaqi Tang Chenyi Huang Feiyang Hao Zhouhui Lian VLM 51 0 0 13 Mar 2025
Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editing Hao Fei Shengqiong Wu H. Zhang Tat-Seng Chua Shuicheng Yan 51 35 0 31 Dec 2024
Towards Unifying Understanding and Generation in the Era of Vision Foundation Models: A Survey from the Autoregression Perspective Shenghao Xie Wenqiang Zu Mingyang Zhao Duo Su Shilong Liu Ruohua Shi Guoqi Li Shanghang Zhang Lei Ma LRM 38 3 0 29 Oct 2024
What Factors Affect Multi-Modal In-Context Learning? An In-Depth Exploration L. Qin Qiguang Chen Hao Fei Zhi Chen Min Li Wanxiang Che 26 5 0 27 Oct 2024
Synergistic Dual Spatial-aware Generation of Image-to-Text and Text-to-Image Yu Zhao Hao Fei Xiangtai Li L. Qin Jiayi Ji Hongyuan Zhu Meishan Zhang M. Zhang Jianguo Wei DiffM 18 1 0 20 Oct 2024
Towards Unified Multimodal Editing with Enhanced Knowledge Collaboration Kaihang Pan Zhaoyu Fan Juncheng Li Qifan Yu Hao Fei Siliang Tang Richang Hong Hanwang Zhang Qianru Sun KELM 24 6 0 30 Sep 2024
From Experts to the Public: Governing Multimodal Language Models in Politically Sensitive Video Analysis Tanusree Sharma Yujin Potter Zachary Kilhoffer Yun Huang Dawn Song Yang Wang 46 3 0 15 Sep 2024
PanoSent: A Panoptic Sextuple Extraction Benchmark for Multimodal Conversational Aspect-based Sentiment Analysis Meng Luo Hao Fei Bobo Li Shengqiong Wu Qian Liu Soujanya Poria Erik Cambria M. Lee W. Hsu 26 7 0 18 Aug 2024
OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding Tao Zhang Xiangtai Li Hao Fei Haobo Yuan Shengqiong Wu Shunping Ji Chen Change Loy Shuicheng Yan LRM MLLM VLM 47 44 0 27 Jun 2024
MG-LLaVA: Towards Multi-Granularity Visual Instruction Tuning Xiangyu Zhao Xiangtai Li Haodong Duan Haian Huang Yining Li Kai Chen Hua Yang VLM MLLM 31 10 0 25 Jun 2024
Rotary Position Embedding for Vision Transformer Byeongho Heo Song Park Dongyoon Han Sangdoo Yun 21 8 0 20 Mar 2024
OMG-Seg: Is One Model Good Enough For All Segmentation? Xiangtai Li Haobo Yuan Wei Li Henghui Ding Size Wu Wenwei Zhang Yining Li Kai Chen Chen Change Loy VLM MLLM ViT 61 48 0 18 Jan 2024
Video-LLaVA: Learning United Visual Representation by Alignment Before Projection Bin Lin Yang Ye Bin Zhu Jiaxi Cui Munan Ning Peng Jin Li-ming Yuan VLM MLLM 182 576 0 16 Nov 2023
MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning Jun Chen Deyao Zhu Xiaoqian Shen Xiang Li Zechun Liu Pengchuan Zhang Raghuraman Krishnamoorthi Vikas Chandra Yunyang Xiong Mohamed Elhoseiny MLLM 152 280 0 14 Oct 2023
MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing Kai Zhang Lingbo Mo Wenhu Chen Huan Sun Yu-Chuan Su EGVM 88 235 0 16 Jun 2023
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models Junnan Li Dongxu Li Silvio Savarese Steven C. H. Hoi VLM MLLM 244 4,186 0 30 Jan 2023
Visual Concepts Tokenization Tao Yang Yuwang Wang Yan Lu Nanning Zheng OCL ViT 21 12 0 20 May 2022
LAVT: Language-Aware Vision Transformer for Referring Image Segmentation Zhao Yang Jiaqi Wang Yansong Tang Kai-xiang Chen Hengshuang Zhao Philip H. S. Torr 115 308 0 04 Dec 2021
GraghVQA: Language-Guided Graph Neural Networks for Graph-based Visual Question Answering Weixin Liang Yanhao Jiang Zixuan Liu GNN 29 23 0 20 Apr 2021
Unsupervised Semantic Segmentation by Contrasting Object Mask Proposals Wouter Van Gansbeke Simon Vandenhende Stamatios Georgoulis Luc Van Gool SSL 175 247 0 11 Feb 2021