Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks

17 June 2022

Papers citing "Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks"

50 / 327 papers shown

Title
Tokenize Anything via Prompting Ting Pan Lulu Tang Xinlong Wang Shiguang Shan VLM 18 22 0 14 Dec 2023
Hallucination Augmented Contrastive Learning for Multimodal Large Language Model Chaoya Jiang Haiyang Xu Mengfan Dong Jiaxing Chen Wei Ye Mingshi Yan Qinghao Ye Ji Zhang Fei Huang Shikun Zhang VLM 13 51 0 12 Dec 2023
4M: Massively Multimodal Masked Modeling David Mizrahi Roman Bachmann Ouguzhan Fatih Kar Teresa Yeo Mingfei Gao Afshin Dehghan Amir Zamir MLLM 34 62 0 11 Dec 2023
Genixer: Empowering Multimodal Large Language Models as a Powerful Data Generator Henry Hengyuan Zhao Pan Zhou Mike Zheng Shou MLLM SyDa 33 7 0 11 Dec 2023
Visual Program Distillation: Distilling Tools and Programmatic Reasoning into Vision-Language Models Yushi Hu Otilia Stretcu Chun-Ta Lu Krishnamurthy Viswanathan Kenji Hata Enming Luo Ranjay Krishna Ariel Fuxman VLM LRM MLLM 32 26 0 05 Dec 2023
UPOCR: Towards Unified Pixel-Level OCR Interface Dezhi Peng Zhenhua Yang Jiaxin Zhang Chongyu Liu Yongxin Shi Kai Ding Fengjun Guo Lianwen Jin 21 10 0 05 Dec 2023
Lenna: Language Enhanced Reasoning Detection Assistant Fei Wei Xinyu Zhang Ailing Zhang Bo-Wen Zhang Xiangxiang Chu MLLM LRM 22 21 0 05 Dec 2023
GIVT: Generative Infinite-Vocabulary Transformers Michael Tschannen Cian Eastwood Fabian Mentzer 6 32 0 04 Dec 2023
Hulk: A Universal Knowledge Translator for Human-Centric Tasks Yizhou Wang YiXuan Wu Shixiang Tang Weizhen He Xun Guo ... Lei Bai Rui Zhao Jian Wu Tong He Wanli Ouyang VLM 28 14 0 04 Dec 2023
PixelLM: Pixel Reasoning with Large Multimodal Model Zhongwei Ren Zhicheng Huang Yunchao Wei Yao-Min Zhao Dongmei Fu Jiashi Feng Xiaojie Jin VLM MLLM LRM 28 78 0 04 Dec 2023
Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts Jialin Wu Xia Hu Yaqing Wang Bo Pang Radu Soricut MoE 9 14 0 01 Dec 2023
Manipulating the Label Space for In-Context Classification Haokun Chen Xu Yang Yuhang Huang Zihan Wu Jing Wang Xin Geng VLM 8 2 0 01 Dec 2023
InstructSeq: Unifying Vision Tasks with Instruction-conditioned Multi-modal Sequence Generation Rongyao Fang Shilin Yan Zhaoyang Huang Jingqiu Zhou Hao Tian Jifeng Dai Hongsheng Li MLLM 22 8 0 30 Nov 2023
X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoning Artemis Panagopoulou Le Xue Ning Yu Junnan Li Dongxu Li Shafiq R. Joty Ran Xu Silvio Savarese Caiming Xiong Juan Carlos Niebles VLM MLLM 28 45 0 30 Nov 2023
Do text-free diffusion models learn discriminative visual representations? Soumik Mukhopadhyay M. Gwilliam Yosuke Yamaguchi Vatsal Agarwal Namitha Padmanabhan Archana Swaminathan Tianyi Zhou Abhinav Shrivastava DiffM 22 11 1 29 Nov 2023
ShapeGPT: 3D Shape Generation with A Unified Multi-modal Language Model Fukun Yin Xin Chen C. Zhang Biao Jiang Zibo Zhao Jiayuan Fan Gang Yu Taihao Li Tao Chen 16 19 0 29 Nov 2023
E-ViLM: Efficient Video-Language Model via Masked Video Modeling with Semantic Vector-Quantized Tokenizer Jacob Zhiyuan Fang Skyler Zheng Vasu Sharma Robinson Piramuthu VLM 30 0 0 28 Nov 2023
Single-Model and Any-Modality for Video Object Tracking Zongwei Wu Jilai Zheng Xiangxuan Ren Florin-Alexandru Vasluianu Chao Ma D. Paudel Luc Van Gool Radu Timofte 46 33 0 27 Nov 2023
Enhancing Visual Grounding and Generalization: A Multi-Task Cycle Training Approach for Vision-Language Models Xiaoyu Yang Lijian Xu Hao Sun Hongsheng Li Shaoting Zhang ObjD 22 4 0 21 Nov 2023
An Embodied Generalist Agent in 3D World Jiangyong Huang Silong Yong Xiaojian Ma Xiongkun Linghu Puhao Li Yan Wang Qing Li Song-Chun Zhu Baoxiong Jia Siyuan Huang LM&Ro 23 131 0 18 Nov 2023
DEED: Dynamic Early Exit on Decoder for Accelerating Encoder-Decoder Transformer Models Peng Tang Pengkai Zhu Tian Li Srikar Appalaraju Vijay Mahadevan R. Manmatha 32 7 0 15 Nov 2023
Vision-Language Instruction Tuning: A Review and Analysis Chen Li Yixiao Ge Dian Li Ying Shan VLM 28 12 0 14 Nov 2023
PerceptionGPT: Effectively Fusing Visual Perception into LLM Renjie Pi Lewei Yao Jiahui Gao Jipeng Zhang Tong Zhang MLLM 18 30 0 11 Nov 2023
Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models Zhang Li Biao Yang Qiang Liu Zhiyin Ma Shuo Zhang Jingxu Yang Yabo Sun Yuliang Liu Xiang Bai MLLM 36 240 0 11 Nov 2023
Analyzing Modular Approaches for Visual Question Decomposition Apoorv Khandelwal Ellie Pavlick Chen Sun 35 4 0 10 Nov 2023
Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks Bin Xiao Haiping Wu Weijian Xu Xiyang Dai Houdong Hu Yumao Lu Michael Zeng Ce Liu Lu Yuan VLM 31 140 0 10 Nov 2023
DAMEX: Dataset-aware Mixture-of-Experts for visual understanding of mixture-of-datasets Yash Jain Harkirat Singh Behl Z. Kira Vibhav Vineet 11 11 0 08 Nov 2023
TEAL: Tokenize and Embed ALL for Multi-modal Large Language Models Zhen Yang Yingxue Zhang Fandong Meng Jie Zhou VLM MLLM 37 3 0 08 Nov 2023
mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration Qinghao Ye Haiyang Xu Jiabo Ye Mingshi Yan Anwen Hu Haowei Liu Qi Qian Ji Zhang Fei Huang Jingren Zhou MLLM VLM 116 367 0 07 Nov 2023
RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketches Jiayuan Gu Sean Kirmani Paul Wohlhart Yao Lu Montse Gonzalez Arenas ... Hao Su Karol Hausman Chelsea Finn Q. Vuong Ted Xiao 22 61 0 03 Nov 2023
LLaVA-Interactive: An All-in-One Demo for Image Chat, Segmentation, Generation and Editing Wei-Ge Chen Irina Spiridonova Jianwei Yang Jianfeng Gao Chun-yue Li MLLM VLM 11 33 0 01 Nov 2023
From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities Md Farhan Ishmam Md Sakib Hossain Shovon M. F. Mridha Nilanjan Dey 35 35 0 01 Nov 2023
Object-centric Video Representation for Long-term Action Anticipation Ce Zhang Changcheng Fu Shijie Wang Nakul Agarwal Kwonjoon Lee Chiho Choi Chen Sun 15 14 0 31 Oct 2023
Grounding Visual Illusions in Language: Do Vision-Language Models Perceive Illusions Like Humans? Yichi Zhang Jiayi Pan Yuchen Zhou Rui Pan Joyce Chai VLM 16 13 0 31 Oct 2023
Exploring Question Decomposition for Zero-Shot VQA Zaid Khan B. Vijaykumar S. Schulter Manmohan Chandraker Yun Fu ReLM 17 9 0 25 Oct 2023
Apollo: Zero-shot MultiModal Reasoning with Multiple Experts Daniela Ben-David Tzuf Paz-Argaman Reut Tsarfaty MoE 21 0 0 25 Oct 2023
CLIP meets Model Zoo Experts: Pseudo-Supervision for Visual Enhancement Mohammadreza Salehi Mehrdad Farajtabar Maxwell Horton Fartash Faghri Hadi Pouransari Raviteja Vemulapalli Oncel Tuzel Ali Farhadi Mohammad Rastegari Sachin Mehta CLIP VLM 22 1 0 21 Oct 2023
Visual Grounding Helps Learn Word Meanings in Low-Data Regimes Chengxu Zhuang Evelina Fedorenko Jacob Andreas 20 10 0 20 Oct 2023
Mastering Robot Manipulation with Multimodal Prompts through Pretraining and Multi-task Fine-tuning Jiachen Li Qiaozi Gao Michael Johnston Xiaofeng Gao Xuehai He Suhaila Shakiah Hangjie Shi R. Ghanadan William Yang Wang LM&Ro 19 12 0 14 Oct 2023
PolyTask: Learning Unified Policies through Behavior Distillation Siddhant Haldar Lerrel Pinto 13 7 0 12 Oct 2023
Ferret: Refer and Ground Anything Anywhere at Any Granularity Haoxuan You Haotian Zhang Zhe Gan Xianzhi Du Bowen Zhang Zirui Wang Liangliang Cao Shih-Fu Chang Yinfei Yang ObjD MLLM VLM 22 298 0 11 Oct 2023
Lightweight In-Context Tuning for Multimodal Unified Models Yixin Chen Shuai Zhang Boran Han Jiaya Jia 11 2 0 08 Oct 2023
VLATTACK: Multimodal Adversarial Attacks on Vision-Language Tasks via Pre-trained Models Ziyi Yin Muchao Ye Tianrong Zhang Tianyu Du Jinguo Zhu Han Liu Jinghui Chen Ting Wang Fenglong Ma AAML VLM CoGe 28 36 0 07 Oct 2023
Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs Shiyu Xuan Qingpei Guo Ming Yang Shiliang Zhang MLLM ObjD 13 38 0 01 Oct 2023
InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists Yulu Gan Sungwoo Park Alexander Schubert Anthony Philippakis Ahmed Alaa VLM 17 21 0 30 Sep 2023
SCOB: Universal Text Understanding via Character-wise Supervised Contrastive Learning with Online Text Rendering for Bridging Domain Gap Daehee Kim Yoon Kim Donghyun Kim Yumin Lim Geewook Kim Taeho Kil 21 3 0 21 Sep 2023
DreamLLM: Synergistic Multimodal Comprehension and Creation Runpei Dong Chunrui Han Yuang Peng Zekun Qi Zheng Ge ... Hao-Ran Wei Xiangwen Kong Xiangyu Zhang Kaisheng Ma Li Yi MLLM 28 168 0 20 Sep 2023
RMT: Retentive Networks Meet Vision Transformers Qihang Fan Huaibo Huang Mingrui Chen Hongmin Liu Ran He ViT 30 73 0 20 Sep 2023
Frequency-Aware Masked Autoencoders for Multimodal Pretraining on Biosignals Ran Liu Ellen L. Zippi Hadi Pouransari Chris Sandino Jingping Nie Hanlin Goh Erdrin Azemi Ali Moin 26 12 0 12 Sep 2023
InstructDiffusion: A Generalist Modeling Interface for Vision Tasks Zigang Geng Binxin Yang Tiankai Hang Chen Li Shuyang Gu ... Jianmin Bao Zheng-Wei Zhang Han Hu Dongdong Chen Baining Guo DiffM VLM 38 92 0 07 Sep 2023