Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic

27 June 2023

Papers citing "Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic"

50 / 114 papers shown

Title
Visual Agents as Fast and Slow Thinkers Guangyan Sun Mingyu Jin Zhenting Wang Cheng-Long Wang Siqi Ma Qifan Wang Ying Nian Wu Ying Nian Wu Dongfang Liu Dongfang Liu LLMAG LRM 77 12 0 16 Aug 2024
CROME: Cross-Modal Adapters for Efficient Multimodal LLM Sayna Ebrahimi Sercan Ö. Arik Tejas Nama Tomas Pfister 37 1 0 13 Aug 2024
Img-Diff: Contrastive Data Synthesis for Multimodal Large Language Models Qirui Jiao Daoyuan Chen Yilun Huang Yaliang Li Ying Shen VLM 27 5 0 08 Aug 2024
Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models Fushuo Huo Wenchao Xu Zhong Zhang Haozhao Wang Zhicheng Chen Peilin Zhao VLM MLLM 61 18 0 04 Aug 2024
GalleryGPT: Analyzing Paintings with Large Multimodal Models Yi Bin Wenhao Shi Yujuan Ding Zhiqiang Hu Zheng Wang Yang Yang See-Kiong Ng H. Shen MLLM 30 11 0 01 Aug 2024
Learning Video Context as Interleaved Multimodal Sequences S. Shao Pengchuan Zhang Y. Li Xide Xia A. Meso Ziteng Gao Jinheng Xie N. Holliman Mike Zheng Shou 41 5 0 31 Jul 2024
ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Models Ming-Kuan Wu Xinyue Cai Jiayi Ji Jiale Li Oucheng Huang Gen Luo Hao Fei Xiaoshuai Sun Rongrong Ji MLLM 40 7 0 31 Jul 2024
Learning Visual Grounding from Generative Vision and Language Model Shijie Wang Dahun Kim A. Taalimi Chen Sun Weicheng Kuo ObjD 32 5 0 18 Jul 2024
Reflective Instruction Tuning: Mitigating Hallucinations in Large Vision-Language Models Jinrui Zhang Teng Wang Haigang Zhang Ping Lu Feng Zheng MLLM LRM VLM 25 3 0 16 Jul 2024
MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMs Yusu Qian Hanrong Ye J. Fauconnier Peter Grasch Yinfei Yang Zhe Gan 108 13 0 01 Jul 2024
$GenderBias-\emph{VL}: Benchmarking Gender Bias in Vision Language Models via Counterfactual Probing$ GenderBias-\emph{VL}: Benchmarking Gender Bias in Vision Language Models via Counterfactual Probing Yisong Xiao Aishan Liu QianJia Cheng Zhenfei Yin Siyuan Liang Jiapeng Li Jing Shao Xianglong Liu Dacheng Tao 33 4 0 30 Jun 2024
LLaRA: Supercharging Robot Learning Data for Vision-Language Policy Xiang Li Cristina Mata J. Park Kumara Kahatapitiya Yoo Sung Jang ... Kanchana Ranasinghe R. Burgert Mu Cai Yong Jae Lee Michael S. Ryoo LM&Ro 62 25 0 28 Jun 2024
Curriculum Learning with Quality-Driven Data Selection Biao Wu Fang Meng Ling-Hao Chen 21 1 0 27 Jun 2024
On Efficient Language and Vision Assistants for Visually-Situated Natural Language Understanding: What Matters in Reading and Reasoning Geewook Kim Minjoon Seo VLM 34 2 0 17 Jun 2024
MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Models Tianle Gu Zeyang Zhou Kexin Huang Dandan Liang Yixu Wang ... Keqing Wang Yujiu Yang Yan Teng Yu Qiao Yingchun Wang ELM 42 9 0 11 Jun 2024
F-LMM: Grounding Frozen Large Multimodal Models Size Wu Sheng Jin Wenwei Zhang Lumin Xu Wentao Liu Wei Li Chen Change Loy MLLM 73 12 0 09 Jun 2024
Towards Semantic Equivalence of Tokenization in Multimodal LLM Shengqiong Wu Hao Fei Xiangtai Li Jiayi Ji Hanwang Zhang Tat-Seng Chua Shuicheng Yan MLLM 59 31 0 07 Jun 2024
VoCoT: Unleashing Visually Grounded Multi-Step Reasoning in Large Multi-Modal Models Zejun Li Ruipu Luo Jiwen Zhang Minghui Qiu Zhongyu Wei Zhongyu Wei LRM MLLM 60 7 0 27 May 2024
User-Friendly Customized Generation with Multi-Modal Prompts Linhao Zhong Yan Hong Wentao Chen Binglin Zhou Yiyi Zhang Jianfu Zhang Liqing Zhang DiffM 35 0 0 26 May 2024
Prompt-Aware Adapter: Towards Learning Adaptive Visual Tokens for Multimodal Large Language Models Yue Zhang Hehe Fan Yi Yang 43 3 0 24 May 2024
Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Models Byung-Kwan Lee Chae Won Kim Beomchan Park Yonghyun Ro MLLM LRM 22 17 0 24 May 2024
V-Zen: Efficient GUI Understanding and Precise Grounding With A Novel Multimodal LLM Abdur Rahman Rajat Chawla Muskaan Kumar Arkajit Datta Adarsh Jha NS Mukunda Ishaan Bhola 40 2 0 24 May 2024
DEEM: Diffusion Models Serve as the Eyes of Large Language Models for Image Perception Run Luo Yunshui Li Longze Chen Wanwei He Ting-En Lin ... Zikai Song Xiaobo Xia Tongliang Liu Min Yang Binyuan Hui VLM DiffM 70 14 0 24 May 2024
Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts Yunxin Li Shenyuan Jiang Baotian Hu Longyue Wang Wanqi Zhong Wenhan Luo Lin Ma Min-Ling Zhang MoE 34 28 0 18 May 2024
Hallucination of Multimodal Large Language Models: A Survey Zechen Bai Pichao Wang Tianjun Xiao Tong He Zongbo Han Zheng Zhang Mike Zheng Shou VLM LRM 80 139 0 29 Apr 2024
UrbanCross: Enhancing Satellite Image-Text Retrieval with Cross-Domain Adaptation Siru Zhong Xixuan Hao Yibo Yan Ying Zhang Yangqiu Song Yuxuan Liang 35 8 0 22 Apr 2024
Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedback Wenyi Xiao Ziwei Huang Leilei Gan Wanggui He Haoyuan Li Zhelun Yu Hao Jiang Fei Wu Linchao Zhu MLLM 37 22 0 22 Apr 2024
HOI-Ref: Hand-Object Interaction Referral in Egocentric Vision Siddhant Bansal Michael Wray Dima Damen 31 3 0 15 Apr 2024
Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs Keen You Haotian Zhang E. Schoop Floris Weers Amanda Swearngin Jeffrey Nichols Yinfei Yang Zhe Gan MLLM 39 82 0 08 Apr 2024
Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward Ruohong Zhang Liangke Gui Zhiqing Sun Yihao Feng Keyang Xu ... Di Fu Chunyuan Li Alexander G. Hauptmann Yonatan Bisk Yiming Yang MLLM 43 57 0 01 Apr 2024
Constructing Multilingual Visual-Text Datasets Revealing Visual Multilingual Ability of Vision Language Models Jesse Atuhurra Iqra Ali Tatsuya Hiraoka Hidetaka Kamigaito Tomoya Iwakura Taro Watanabe 38 1 0 29 Mar 2024
Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want Weifeng Lin Xinyu Wei Ruichuan An Peng Gao Bocheng Zou Yulin Luo Siyuan Huang Shanghang Zhang Hongsheng Li VLM 58 32 0 29 Mar 2024
RH20T-P: A Primitive-Level Robotic Dataset Towards Composable Generalization Agents Zeren Chen Zhelun Shi Xiaoya Lu Lehan He Sucheng Qian ... Zhen-fei Yin Jing Shao Jing Shao Cewu Lu Cewu Lu 33 5 0 28 Mar 2024
Effectiveness Assessment of Recent Large Vision-Language Models Yao Jiang Xinyu Yan Ge-Peng Ji Keren Fu Meijun Sun Huan Xiong Deng-Ping Fan Fahad Shahbaz Khan 27 14 0 07 Mar 2024
LORS: Low-rank Residual Structure for Parameter-Efficient Network Stacking Jialin Li Qiang Nie Weifu Fu Yuhuan Lin Guangpin Tao Yong-Jin Liu Chengjie Wang 25 4 0 07 Mar 2024
BBA: Bi-Modal Behavioral Alignment for Reasoning with Large Vision-Language Models Xueliang Zhao Xinting Huang Tingchen Fu Qintong Li Shansan Gong Lemao Liu Wei Bi Lingpeng Kong LRM 33 1 0 21 Feb 2024
CoLLaVO: Crayon Large Language and Vision mOdel Byung-Kwan Lee Beomchan Park Chae Won Kim Yonghyun Ro VLM MLLM 24 16 0 17 Feb 2024
SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models Chris Liu Renrui Zhang Longtian Qiu Siyuan Huang Weifeng Lin ... Hao Shao Pan Lu Hongsheng Li Yu Qiao Peng Gao MLLM 126 106 0 08 Feb 2024
LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMs Shaoxiang Chen Zequn Jie Lin Ma MoE 38 46 0 29 Jan 2024
GroundingGPT:Language Enhanced Multi-modal Grounding Model Zhaowei Li Qi Xu Dong Zhang Hang Song Yiqing Cai ... Junting Pan Zefeng Li Van Tu Vu Zhida Huang Tao Wang 18 37 0 11 Jan 2024
LISA++: An Improved Baseline for Reasoning Segmentation with Large Language Model Senqiao Yang Tianyuan Qu Xin Lai Zhuotao Tian Bohao Peng Shu-Lin Liu Jiaya Jia VLM 21 28 0 28 Dec 2023
Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Model Shraman Pramanick Guangxing Han Rui Hou Sayan Nag Ser-Nam Lim Nicolas Ballas Qifan Wang Rama Chellappa Amjad Almahairi VLM MLLM 38 29 0 19 Dec 2023
M3DBench: Let's Instruct Large Models with Multi-modal 3D Prompts Mingsheng Li Xin Chen C. Zhang Sijin Chen Hongyuan Zhu Fukun Yin Gang Yu Tao Chen 17 23 0 17 Dec 2023
See, Say, and Segment: Teaching LMMs to Overcome False Premises Tsung-Han Wu Giscard Biamby David M. Chan Lisa Dunlap Ritwik Gupta Xudong Wang Joseph E. Gonzalez Trevor Darrell VLM MLLM 30 18 0 13 Dec 2023
LMDrive: Closed-Loop End-to-End Driving with Large Language Models Hao Shao Yuxuan Hu Letian Wang Steven L. Waslander Yu Liu Hongsheng Li ELM 30 110 0 12 Dec 2023
MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perception Yiran Qin Enshen Zhou Qichang Liu Zhen-fei Yin Lu Sheng Ruimao Zhang Yu Qiao Jing Shao LM&Ro 20 39 0 12 Dec 2023
Dolphins: Multimodal Language Model for Driving Yingzi Ma Yulong Cao Jiachen Sun Marco Pavone Chaowei Xiao MLLM 21 49 0 01 Dec 2023
InstructSeq: Unifying Vision Tasks with Instruction-conditioned Multi-modal Sequence Generation Rongyao Fang Shilin Yan Zhaoyang Huang Jingqiu Zhou Hao Tian Jifeng Dai Hongsheng Li MLLM 28 8 0 30 Nov 2023
Text as Images: Can Multimodal Large Language Models Follow Printed Instructions in Pixels? Xiujun Li Yujie Lu Zhe Gan Jianfeng Gao William Yang Wang Yejin Choi VLM MLLM 28 1 0 29 Nov 2023
LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models Yanwei Li Chengyao Wang Jiaya Jia VLM MLLM 36 259 0 28 Nov 2023