Visual Instruction Tuning

17 April 2023

Papers citing "Visual Instruction Tuning"

50 / 2,159 papers shown

Title
On Speculative Decoding for Multimodal Large Language Models Mukul Gagrani Raghavv Goel Wonseok Jeon Junyoung Park Mingu Lee Christopher Lott LRM 27 7 0 13 Apr 2024
Enhancing Visual Question Answering through Question-Driven Image Captions as Prompts Övgü Özdemir Erdem Akagündüz 36 10 0 12 Apr 2024
FGAIF: Aligning Large Vision-Language Models with Fine-grained AI Feedback Liqiang Jing Xinya Du 71 17 0 07 Apr 2024
Koala: Key frame-conditioned long video-LLM Reuben Tan Ximeng Sun Ping Hu Jui-hsien Wang Hanieh Deilamsalehy Bryan A. Plummer Bryan C. Russell Kate Saenko 38 35 0 05 Apr 2024
Physical Property Understanding from Language-Embedded Feature Fields Albert J. Zhai Yuan Shen Emily Y. Chen Gloria X. Wang Xinlei Wang Sheng Wang Kaiyu Guan Shenlong Wang 33 13 0 05 Apr 2024
ZeroCAP: Zero-Shot Multi-Robot Context Aware Pattern Formation via Large Language Models Vishnunandan L. N. Venkatesh Byung-Cheol Min LM&Ro 66 2 0 02 Apr 2024
Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward Ruohong Zhang Liangke Gui Zhiqing Sun Yihao Feng Keyang Xu ... Di Fu Chunyuan Li Alexander G. Hauptmann Yonatan Bisk Yiming Yang MLLM 46 57 0 01 Apr 2024
Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want Weifeng Lin Xinyu Wei Ruichuan An Peng Gao Bocheng Zou Yulin Luo Siyuan Huang Shanghang Zhang Hongsheng Li VLM 58 32 0 29 Mar 2024
RH20T-P: A Primitive-Level Robotic Dataset Towards Composable Generalization Agents Zeren Chen Zhelun Shi Xiaoya Lu Lehan He Sucheng Qian ... Zhen-fei Yin Jing Shao Jing Shao Cewu Lu Cewu Lu 33 5 0 28 Mar 2024
RL for Consistency Models: Faster Reward Guided Text-to-Image Generation Owen Oertell Jonathan D. Chang Yiyi Zhang Kianté Brantley Wen Sun EGVM 36 4 0 25 Mar 2024
Surgical-LVLM: Learning to Adapt Large Vision-Language Model for Grounded Visual Question Answering in Robotic Surgery Guan-Feng Wang Long Bai Wan Jun Nah Jie Wang Zhaoxi Zhang Zhen Chen Jinlin Wu Mobarakol Islam Hongbin Liu Hongliang Ren 40 14 0 22 Mar 2024
Prioritized Semantic Learning for Zero-shot Instance Navigation Xander Sun Louis Lau Hoyard Zhi Ronghe Qiu Junwei Liang 30 8 0 18 Mar 2024
Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Models Yifan Li Hangyu Guo Kun Zhou Wayne Xin Zhao Ji-Rong Wen 50 38 0 14 Mar 2024
UniCode: Learning a Unified Codebook for Multimodal Large Language Models Sipeng Zheng Bohan Zhou Yicheng Feng Ye Wang Zongqing Lu VLM MLLM 31 7 0 14 Mar 2024
DialogGen: Multi-modal Interactive Dialogue System for Multi-turn Text-to-Image Generation Minbin Huang Yanxin Long Xinchi Deng Ruihang Chu Jiangfeng Xiong Xiaodan Liang Hong Cheng Qinglin Lu Wei Liu MLLM EGVM 59 8 0 13 Mar 2024
XPSR: Cross-modal Priors for Diffusion-based Image Super-Resolution Yunpeng Qu Kun Yuan Kai Zhao Qizhi Xie Jinhua Hao Ming-hui Sun Chao Zhou 27 16 0 08 Mar 2024
Yi: Open Foundation Models by 01.AI 01. AI Alex Young 01.AI Alex Young Bei Chen Chao Li ... Yue Wang Yuxuan Cai Zhenyu Gu Zhiyuan Liu Zonghong Dai OSLM LRM 121 495 0 07 Mar 2024
Effectiveness Assessment of Recent Large Vision-Language Models Yao Jiang Xinyu Yan Ge-Peng Ji Keren Fu Meijun Sun Huan Xiong Deng-Ping Fan Fahad Shahbaz Khan 27 14 0 07 Mar 2024
On the Essence and Prospect: An Investigation of Alignment Approaches for Big Models Xinpeng Wang Shitong Duan Xiaoyuan Yi Jing Yao Shanlin Zhou Zhihua Wei Peng Zhang Dongkuan Xu Maosong Sun Xing Xie OffRL 33 16 0 07 Mar 2024
CoTBal: Comprehensive Task Balancing for Multi-Task Visual Instruction Tuning Yanqi Dai Dong Jing Nanyi Fei Zhiwu Lu Nanyi Fei Guoxing Yang Zhiwu Lu 48 3 0 07 Mar 2024
Here Comes The AI Worm: Unleashing Zero-click Worms that Target GenAI-Powered Applications Stav Cohen Ron Bitton Ben Nassi 30 17 0 05 Mar 2024
Beyond Specialization: Assessing the Capabilities of MLLMs in Age and Gender Estimation Maksim Kuprashevich Grigorii Alekseenko Irina Tolstykh ELM 48 4 0 04 Mar 2024
UNITS: A Unified Multi-Task Time Series Model Shanghua Gao Teddy Koker Owen Queen Thomas Hartvigsen Theodoros Tsiligkaridis Marinka Zitnik AI4TS 34 15 0 29 Feb 2024
Exploring the Potential of Large Language Models for Improving Digital Forensic Investigation Efficiency Akila Wickramasekara F. Breitinger Mark Scanlon 42 7 0 29 Feb 2024
Unveiling Typographic Deceptions: Insights of the Typographic Vulnerability in Large Vision-Language Model Hao-Ran Cheng Erjia Xiao Jindong Gu Le Yang Jinhao Duan Jize Zhang Jiahang Cao Kaidi Xu Renjing Xu 29 6 0 29 Feb 2024
From Summary to Action: Enhancing Large Language Models for Complex Tasks with Open World APIs Yulong Liu Yunlong Yuan Chunwei Wang Jianhua Han Yongqiang Ma Li Zhang Nanning Zheng Hang Xu LLMAG 24 5 0 28 Feb 2024
Polos: Multimodal Metric Learning from Human Feedback for Image Captioning Yuiga Wada Kanta Kaneda Daichi Saito Komei Sugiura 29 24 0 28 Feb 2024
A Cognitive Evaluation Benchmark of Image Reasoning and Description for Large Vision-Language Models Xiujie Song Mengyue Wu Ke Zhu Chunhao Zhang Yanyi Chen LRM ELM 31 3 0 28 Feb 2024
Diffusion Model-Based Image Editing: A Survey Yi Huang Jiancheng Huang Yifan Liu Mingfu Yan Jiaxi Lv Jianzhuang Liu Wei Xiong He Zhang Liangliang Cao Liangliang Cao EGVM 66 84 0 27 Feb 2024
AutoMMLab: Automatically Generating Deployable Models from Language Instructions for Computer Vision Tasks Zekang Yang Wang Zeng Sheng Jin Chao Qian Ping Luo Wentao Liu MLLM VLM 46 8 0 23 Feb 2024
Mitigating Fine-tuning based Jailbreak Attack with Backdoor Enhanced Safety Alignment Jiong Wang Jiazhao Li Yiquan Li Xiangyu Qi Junjie Hu Yixuan Li P. McDaniel Muhao Chen Bo Li Chaowei Xiao AAML SILM 32 16 0 22 Feb 2024
PALO: A Polyglot Large Multimodal Model for 5B People Muhammad Maaz H. Rasheed Abdelrahman M. Shaker Salman Khan Hisham Cholakal Rao M. Anwer Timothy Baldwin M. Felsberg Fahad S. Khan VLM LRM 67 13 0 22 Feb 2024
We Choose to Go to Space: Agent-driven Human and Multi-Robot Collaboration in Microgravity Miao Xin Zhongrui You Zihan Zhang Taoran Jiang Tingjia Xu Haotian Liang Guojing Ge Yuchen Ji Shentong Mo Jian Cheng 35 0 0 22 Feb 2024
Subobject-level Image Tokenization Delong Chen Samuel Cahyawijaya Jianfeng Liu Baoyuan Wang Pascale Fung VLM OCL 46 6 0 22 Feb 2024
On Large Visual Language Models for Medical Imaging Analysis: An Empirical Study Minh-Hao Van Prateek Verma Xintao Wu LM&MA 36 29 0 21 Feb 2024
VL-Trojan: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language Models Jiawei Liang Siyuan Liang Man Luo Aishan Liu Dongchen Han Ee-Chien Chang Xiaochun Cao 38 37 0 21 Feb 2024
SInViG: A Self-Evolving Interactive Visual Agent for Human-Robot Interaction Jie Xu Hanbo Zhang Xinghang Li Huaping Liu Xuguang Lan Tao Kong LM&Ro 25 3 0 19 Feb 2024
RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Model Jianhao Yuan Shuyang Sun Daniel Omeiza Bo-Lu Zhao Paul Newman Lars Kunze Matthew Gadd LRM 16 47 0 16 Feb 2024
Exploring Perceptual Limitation of Multimodal Large Language Models Jiarui Zhang Jinyi Hu Mahyar Khayatkhoei Filip Ilievski Maosong Sun LRM 24 10 0 12 Feb 2024
LLMs for Coding and Robotics Education Peng Shu Huaqin Zhao Hanqi Jiang Yiwei Li Shaochen Xu ... Zheng Liu Guoyu Lu Le Guan Gong Chen Xianqiao Wang Tianming Liu 30 5 0 09 Feb 2024
Large Language Models: A Survey Shervin Minaee Tomáš Mikolov Narjes Nikzad M. Asgari-Chenaghlu R. Socher Xavier Amatriain Jianfeng Gao ALM LM&MA ELM 120 353 0 09 Feb 2024
CREMA: Generalizable and Efficient Video-Language Reasoning via Multimodal Modular Fusion Shoubin Yu Jaehong Yoon Mohit Bansal 77 4 0 08 Feb 2024
SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models Chris Liu Renrui Zhang Longtian Qiu Siyuan Huang Weifeng Lin ... Hao Shao Pan Lu Hongsheng Li Yu Qiao Peng Gao MLLM 126 106 0 08 Feb 2024
Dual-View Visual Contextualization for Web Navigation Jihyung Kil Chan Hee Song Boyuan Zheng Xiang Deng Yu-Chuan Su Wei-Lun Chao EgoV 22 12 0 06 Feb 2024
Visual Text Meets Low-level Vision: A Comprehensive Survey on Visual Text Processing Yan Shu Weichao Zeng Zhenhang Li Fangmin Zhao Yu Zhou 28 3 0 05 Feb 2024
Towards Urban General Intelligence: A Review and Outlook of Urban Foundation Models Weijiao Zhang Jindong Han Zhao Xu Hang Ni Hao Liu Hui Xiong Hui Xiong AI4CE 77 15 0 30 Jan 2024
LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMs Shaoxiang Chen Zequn Jie Lin Ma MoE 38 46 0 29 Jan 2024
Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception Junyang Wang Haiyang Xu Jiabo Ye Mingshi Yan Weizhou Shen Ji Zhang Fei Huang Jitao Sang 24 103 0 29 Jan 2024
Towards 3D Molecule-Text Interpretation in Language Models Sihang Li Zhiyuan Liu Yancheng Luo Xiang Wang Xiangnan He Kenji Kawaguchi Tat-Seng Chua Qi Tian AI4CE 24 42 0 25 Jan 2024
CCA: Collaborative Competitive Agents for Image Editing Tiankai Hang Shuyang Gu Dong Chen Xin Geng Baining Guo 20 5 0 23 Jan 2024