Visual Instruction Tuning

17 April 2023

Papers citing "Visual Instruction Tuning"

50 / 3,278 papers shown

Title
Gen4Gen: Generative Data Pipeline for Generative Multi-Concept Composition Chun-Hsiao Yeh Ta-Ying Cheng He-Yen Hsieh Chuan-En Lin Yi Ma Andrew Markham Niki Trigoni H. T. Kung Yubei Chen DiffM 25 4 0 23 Feb 2024
AutoMMLab: Automatically Generating Deployable Models from Language Instructions for Computer Vision Tasks Zekang Yang Wang Zeng Sheng Jin Chao Qian Ping Luo Wentao Liu MLLM VLM 66 8 0 23 Feb 2024
Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Control Masatoshi Uehara Yulai Zhao Kevin Black Ehsan Hajiramezanali Gabriele Scalia N. Diamant Alex Tseng Tommaso Biancalani Sergey Levine 47 42 0 23 Feb 2024
Mitigating Fine-tuning based Jailbreak Attack with Backdoor Enhanced Safety Alignment Jiong Wang Jiazhao Li Yiquan Li Xiangyu Qi Junjie Hu Yixuan Li P. McDaniel Muhao Chen Bo Li Chaowei Xiao AAML SILM 40 18 0 22 Feb 2024
PALO: A Polyglot Large Multimodal Model for 5B People Muhammad Maaz H. Rasheed Abdelrahman M. Shaker Salman Khan Hisham Cholakal Rao M. Anwer Timothy Baldwin Michael Felsberg Fahad S. Khan VLM LRM 85 13 0 22 Feb 2024
DualFocus: Integrating Macro and Micro Perspectives in Multi-modal Large Language Models Yuhang Cao Pan Zhang Xiao-wen Dong Dahua Lin Jiaqi Wang 45 11 0 22 Feb 2024
LLMBind: A Unified Modality-Task Integration Framework Bin Zhu Munan Ning Peng Jin Bin Lin Jinfa Huang ... Junwu Zhang Zhenyu Tang Mingjun Pan Xing Zhou Li-ming Yuan MLLM 40 6 0 22 Feb 2024
Uncertainty-Aware Evaluation for Vision-Language Models Vasily Kostumov Bulat Nutfullin Oleg Pilipenko Eugene Ilyushin ELM 55 8 0 22 Feb 2024
We Choose to Go to Space: Agent-driven Human and Multi-Robot Collaboration in Microgravity Miao Xin Zhongrui You Zihan Zhang Taoran Jiang Tingjia Xu Haotian Liang Guojing Ge Yuchen Ji Shentong Mo Jian Cheng 37 0 0 22 Feb 2024
Subobject-level Image Tokenization Delong Chen Samuel Cahyawijaya Jianfeng Liu Baoyuan Wang Pascale Fung VLM OCL 60 7 0 22 Feb 2024
On Large Visual Language Models for Medical Imaging Analysis: An Empirical Study Minh-Hao Van Prateek Verma Xintao Wu LM&MA 49 30 0 21 Feb 2024
DeiSAM: Segment Anything with Deictic Prompting Hikaru Shindo Manuel Brack Gopika Sudhakaran Devendra Singh Dhami P. Schramowski Kristian Kersting VLM 46 2 0 21 Feb 2024
VL-Trojan: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language Models Jiawei Liang Siyuan Liang Man Luo Aishan Liu Dongchen Han Ee-Chien Chang Xiaochun Cao 47 38 0 21 Feb 2024
Data-driven Discovery with Large Generative Models Bodhisattwa Prasad Majumder Harshit Surana Dhruv Agarwal Sanchaita Hazra Ashish Sabharwal Peter Clark 48 9 0 21 Feb 2024
The Wolf Within: Covert Injection of Malice into MLLM Societies via an MLLM Operative Zhen Tan Chengshuai Zhao Raha Moraffah Yifan Li Yu Kong Tianlong Chen Huan Liu 44 15 0 20 Feb 2024
Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions Akash Ghosh Arkadeep Acharya Sriparna Saha Vinija Jain Aman Chadha VLM 62 26 0 20 Feb 2024
A Touch, Vision, and Language Dataset for Multimodal Alignment Letian Fu Gaurav Datta Huang Huang Will Panitch Jaimyn Drake Joseph Ortiz Mustafa Mukadam Mike Lambeta Roberto Calandra Ken Goldberg VLM 40 34 0 20 Feb 2024
How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Prompts Yusu Qian Haotian Zhang Yinfei Yang Zhe Gan 100 26 0 20 Feb 2024
A Survey on Knowledge Distillation of Large Language Models Xiaohan Xu Ming Li Chongyang Tao Tao Shen Reynold Cheng Jinyang Li Can Xu Dacheng Tao Dinesh Manocha KELM VLM 46 104 0 20 Feb 2024
Slot-VLM: SlowFast Slots for Video-Language Modeling Jiaqi Xu Cuiling Lan Wenxuan Xie Xuejin Chen Yan Lu MLLM VLM 35 7 0 20 Feb 2024
Reflect-RL: Two-Player Online RL Fine-Tuning for LMs Runlong Zhou Simon S. Du Beibin Li OffRL 49 3 0 20 Feb 2024
Generative AI Security: Challenges and Countermeasures Banghua Zhu Norman Mu Jiantao Jiao David Wagner AAML SILM 66 8 0 20 Feb 2024
Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models Christian Schlarmann Naman D. Singh Francesco Croce Matthias Hein VLM AAML 52 39 0 19 Feb 2024
Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Models Didi Zhu Zhongyi Sun Zexi Li Tao Shen Ke Yan Shouhong Ding Kun Kuang Chao Wu CLL KELM MoMe 77 25 0 19 Feb 2024
Learning the Unlearned: Mitigating Feature Suppression in Contrastive Learning Jihai Zhang Xiang Lan Xiaoye Qu Yu Cheng Mengling Feng Bryan Hooi SSL 24 4 0 19 Feb 2024
SInViG: A Self-Evolving Interactive Visual Agent for Human-Robot Interaction Jie Xu Hanbo Zhang Xinghang Li Huaping Liu Xuguang Lan Tao Kong LM&Ro 38 3 0 19 Feb 2024
Efficient Multimodal Learning from Data-centric Perspective Muyang He Yexin Liu Boya Wu Jianhao Yuan Yueze Wang Tiejun Huang Bo Zhao MLLM 38 85 0 18 Feb 2024
Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoning Long Qian Juncheng Billy Li Yu-hao Wu Yaobo Ye Hao Fei Tat-Seng Chua Yueting Zhuang Siliang Tang MLLM LRM 65 47 0 18 Feb 2024
Aligning Modalities in Vision Large Language Models via Preference Fine-tuning Yiyang Zhou Chenhang Cui Rafael Rafailov Chelsea Finn Huaxiu Yao VLM MLLM 43 89 0 18 Feb 2024
ChatEarthNet: A Global-Scale Image-Text Dataset Empowering Vision-Language Geo-Foundation Models Zhenghang Yuan Zhitong Xiong Lichao Mou Xiao Xiang Zhu 33 8 0 17 Feb 2024
PaLM2-VAdapter: Progressively Aligned Language Model Makes a Strong Vision-language Adapter Junfei Xiao Zheng Xu Alan Yuille Shen Yan Boyu Wang 33 3 0 16 Feb 2024
RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Model Jianhao Yuan Shuyang Sun Daniel Omeiza Bo Zhao Paul Newman Lars Kunze Matthew Gadd LRM 41 50 0 16 Feb 2024
Using Left and Right Brains Together: Towards Vision and Language Planning Jun Cen Chenfei Wu Xiao Liu Sheng-Siang Yin Yixuan Pei Jinglong Yang Qifeng Chen Nan Duan Jianguo Zhang 68 3 0 16 Feb 2024
Analyzing the Roles of Language and Vision in Learning from Limited Data Allison Chen Ilia Sucholutsky Olga Russakovsky Thomas Griffiths VLM 29 2 0 15 Feb 2024
DoRA: Weight-Decomposed Low-Rank Adaptation Shih-yang Liu Chien-Yi Wang Hongxu Yin Pavlo Molchanov Yu-Chiang Frank Wang Kwang-Ting Cheng Min-Hung Chen 47 345 0 14 Feb 2024
OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM Yutao Hu Tian-Xin Li Quanfeng Lu Wenqi Shao Junjun He Yu Qiao Ping Luo ELM LM&MA 37 52 0 14 Feb 2024
Pretraining Vision-Language Model for Difference Visual Question Answering in Longitudinal Chest X-rays Yeongjae Cho Taehee Kim Heejun Shin Sungzoon Cho Dongmyung Shin 15 2 0 14 Feb 2024
Open-Vocabulary Segmentation with Unpaired Mask-Text Supervision Zhaoqing Wang Xiaobo Xia Ziye Chen Xiao He Yandong Guo Biwei Huang Tongliang Liu VLM 31 11 0 14 Feb 2024
Interpretable Measures of Conceptual Similarity by Complexity-Constrained Descriptive Auto-Encoding Alessandro Achille Greg Ver Steeg Tian Yu Liu Matthew Trager Carson Klingenberg Stefano Soatto 38 1 0 14 Feb 2024
Mitigating Object Hallucination in Large Vision-Language Models via Classifier-Free Guidance Linxi Zhao Yihe Deng Weitong Zhang Quanquan Gu MLLM 28 30 0 13 Feb 2024
Test-Time Backdoor Attacks on Multimodal Large Language Models Dong Lu Tianyu Pang Chao Du Qian Liu Xianjun Yang Min Lin AAML 63 21 0 13 Feb 2024
Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast Xiangming Gu Xiaosen Zheng Tianyu Pang Chao Du Qian Liu Ye Wang Jing Jiang Min Lin LLMAG LM&Ro 37 50 0 13 Feb 2024
Intriguing Differences Between Zero-Shot and Systematic Evaluations of Vision-Language Transformer Models Shaeke Salman M. Shams Xiuwen Liu Lingjiong Zhu VLM 27 2 0 13 Feb 2024
Visual Question Answering Instruction: Unlocking Multimodal Large Language Model To Domain-Specific Visual Multitasks Jusung Lee Sungguk Cha Younghyun Lee Cheoljong Yang MLLM LRM 32 7 0 13 Feb 2024
Visually Dehallucinative Instruction Generation Sungguk Cha Jusung Lee Younghyun Lee Cheoljong Yang MLLM 22 5 0 13 Feb 2024
THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation Wilbert Pumacay Ishika Singh Jiafei Duan Ranjay Krishna Jesse Thomason Dieter Fox 29 40 0 13 Feb 2024
LLaGA: Large Language and Graph Assistant Runjin Chen Tong Zhao Ajay Jaiswal Neil Shah Zhangyang Wang 23 57 0 13 Feb 2024
World Model on Million-Length Video And Language With Blockwise RingAttention Hao Liu Wilson Yan Matei A. Zaharia Pieter Abbeel VGen 44 64 0 13 Feb 2024
Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models Siddharth Karamcheti Suraj Nair Ashwin Balakrishna Percy Liang Thomas Kollar Dorsa Sadigh MLLM VLM 59 104 0 12 Feb 2024
G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering Xiaoxin He Yijun Tian Yifei Sun Nitesh Chawla T. Laurent Yann LeCun Xavier Bresson Bryan Hooi RALM 119 73 0 12 Feb 2024