Visual Instruction Tuning

17 April 2023

Papers citing "Visual Instruction Tuning"

50 / 3,227 papers shown

Title
InstructDET: Diversifying Referring Object Detection with Generalized Instructions Ronghao Dang Jiangyan Feng Haodong Zhang Chongjian Ge Lin Song ... Chengju Liu Qi Chen Feng Zhu Rui Zhao Yibing Song ObjD 27 11 0 08 Oct 2023
UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model Jiabo Ye Anwen Hu Haiyang Xu Qinghao Ye Mingshi Yan ... Ji Zhang Qin Jin Liang He Xin Lin Feiyan Huang VLM MLLM 123 84 0 08 Oct 2023
AvalonBench: Evaluating LLMs Playing the Game of Avalon Jonathan Light Min Cai Sheng Shen Ziniu Hu LLMAG 17 0 0 08 Oct 2023
Compositional Semantics for Open Vocabulary Spatio-semantic Representations Robin Karlsson Francisco Lepe-Salazar K. Takeda VLM 50 1 0 08 Oct 2023
HowToCaption: Prompting LLMs to Transform Video Annotations at Scale Nina Shvetsova Anna Kukleva Xudong Hong Christian Rupprecht Bernt Schiele Hilde Kuehne 45 25 0 07 Oct 2023
ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations Yue Jiang E. Schoop Amanda Swearngin Jeffrey Nichols MLLM 18 17 0 07 Oct 2023
Exploring the Potential of Multi-Modal AI for Driving Hazard Prediction Korawat Charoenpitaks Van-Quang Nguyen Masanori Suganuma Masahiro Takahashi Ryoma Niihara Takayuki Okatani 25 1 0 07 Oct 2023
Improved Baselines with Visual Instruction Tuning Haotian Liu Chunyuan Li Yuheng Li Yong Jae Lee VLM MLLM 43 2,429 0 05 Oct 2023
Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! Xiangyu Qi Yi Zeng Tinghao Xie Pin-Yu Chen Ruoxi Jia Prateek Mittal Peter Henderson SILM 44 524 0 05 Oct 2023
On the Performance of Multimodal Language Models Utsav Garg Erhan Bas MLLM 11 0 0 04 Oct 2023
Misusing Tools in Large Language Models With Visual Adversarial Examples Xiaohan Fu Zihan Wang Shuheng Li Rajesh K. Gupta Niloofar Mireshghallah Taylor Berg-Kirkpatrick Earlence Fernandes AAML 29 24 0 04 Oct 2023
Multimodal Question Answering for Unified Information Extraction Yuxuan Sun Kai Zhang Yu-Chuan Su 32 8 0 04 Oct 2023
ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language Models Yi-Lin Sung Jaehong Yoon Mohit Bansal VLM 17 14 0 04 Oct 2023
ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks Zejun Li Ye Wang Mengfei Du Qingwen Liu Binhao Wu ... Zhihao Fan Jie Fu Jingjing Chen Xuanjing Huang Zhongyu Wei 27 13 0 04 Oct 2023
MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts Pan Lu Hritik Bansal Tony Xia Jiacheng Liu Chun-yue Li Hannaneh Hajishirzi Hao Cheng Kai-Wei Chang Michel Galley Jianfeng Gao LRM MLLM 43 496 0 03 Oct 2023
Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Driving Tushar Choudhary Vikrant Dewangan Shivam Chandhok Shubham Priyadarshan Anushka Jain A. K. Singh Siddharth Srivastava Krishna Murthy Jatavallabhula K. M. Krishna 50 58 0 03 Oct 2023
Towards End-to-End Embodied Decision Making via Multi-modal Large Language Model: Explorations with GPT4-Vision and Beyond Liang Chen Yichi Zhang Shuhuai Ren Haozhe Zhao Zefan Cai Yuchi Wang Peiyi Wang Tianyu Liu Baobao Chang LM&Ro LLMAG 33 41 0 03 Oct 2023
Tuning Large language model for End-to-end Speech Translation Hao Zhang Nianwen Si Yaqi Chen Wenlin Zhang Xu Yang Dan Qu Xiaolin Jiao 15 8 0 03 Oct 2023
Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving Long Chen Oleg Sinavski Jan Hünermann Alice Karnsund Andrew James Willmott Danny Birch Daniel Maund Jamie Shotton MLLM 15 180 0 03 Oct 2023
HallE-Control: Controlling Object Hallucination in Large Multimodal Models Bohan Zhai Shijia Yang Chenfeng Xu Sheng Shen Kurt Keutzer Chunyuan Li Manling Li MLLM 23 12 0 03 Oct 2023
Fool Your (Vision and) Language Model With Embarrassingly Simple Permutations Yongshuo Zong Tingyang Yu Ruchika Chavhan Bingchen Zhao Timothy M. Hospedales MLLM AAML LRM 27 18 0 02 Oct 2023
DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model Zhenhua Xu Yujia Zhang Enze Xie Zhen Zhao Yong Guo Kwan-Yee. K. Wong Zhenguo Li Hengshuang Zhao MLLM 22 251 0 02 Oct 2023
Making LLaMA SEE and Draw with SEED Tokenizer Yuying Ge Sijie Zhao Ziyun Zeng Yixiao Ge Chen Li Xintao Wang Ying Shan 32 128 0 02 Oct 2023
GRID: A Platform for General Robot Intelligence Development Sai H. Vemprala Shuhang Chen Abhinav Shukla Dinesh Narayanan Ashish Kapoor 25 10 0 02 Oct 2023
Application of frozen large-scale models to multimodal task-oriented dialogue Tatsuki Kawamoto Takuma Suzuki Ko Miyama Takumi Meguro Tomohiro Takagi 27 0 0 02 Oct 2023
Analyzing and Mitigating Object Hallucination in Large Vision-Language Models Yiyang Zhou Chenhang Cui Jaehong Yoon Linjun Zhang Zhun Deng Chelsea Finn Mohit Bansal Huaxiu Yao MLLM 34 162 0 01 Oct 2023
Comics for Everyone: Generating Accessible Text Descriptions for Comic Strips Reshma Ramaprasad 6 5 0 01 Oct 2023
Reformulating Vision-Language Foundation Models and Datasets Towards Universal Multimodal Assistants Tianyu Yu Jinyi Hu Yuan Yao Haoye Zhang Yue Zhao ... Jiao Xue Dahai Li Zhiyuan Liu Hai-Tao Zheng Maosong Sun VLM MLLM 25 19 0 01 Oct 2023
Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learning Mustafa Shukor Alexandre Ramé Corentin Dancette Matthieu Cord LRM MLLM 38 20 0 01 Oct 2023
Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs Shiyu Xuan Qingpei Guo Ming Yang Shiliang Zhang MLLM ObjD 18 38 0 01 Oct 2023
PixArt- $α$ : Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis Junsong Chen Jincheng Yu Chongjian Ge Lewei Yao Enze Xie ... Zhongdao Wang James T. Kwok Ping Luo Huchuan Lu Zhenguo Li DiffM 28 387 0 30 Sep 2023
AutomaTikZ: Text-Guided Synthesis of Scientific Vector Graphics with TikZ Jonas Belouadi Anne Lauscher Steffen Eger 21 27 0 30 Sep 2023
Self-Specialization: Uncovering Latent Expertise within Large Language Models Junmo Kang Hongyin Luo Yada Zhu Jacob A. Hansen James R. Glass David D. Cox Alan Ritter Rogerio Feris Leonid Karlinsky ALM MoMe 21 4 0 29 Sep 2023
CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets Lifan Yuan Yangyi Chen Xingyao Wang Yi Ren Fung Hao Peng Heng Ji LLMAG KELM 27 58 0 29 Sep 2023
Suspicion-Agent: Playing Imperfect Information Games with Theory of Mind Aware GPT-4 Jiaxian Guo Bo Yang Paul D. Yoo Bill Yuchen Lin Yusuke Iwasawa Yutaka Matsuo LLMAG 15 41 0 29 Sep 2023
Guiding Instruction-based Image Editing via Multimodal Large Language Models Johannes Frey Wenze Hu Xianzhi Du William Yang Wang Yinfei Yang Zhe Gan 40 88 0 29 Sep 2023
ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning Yuanyi Zhong Alihusein Kuwajerwala Sacha Morin Krishna Murthy Jatavallabhula Bipasha Sen ... Celso Miguel de Melo Joshua B. Tenenbaum Antonio Torralba Florian Shkurti Liam Paull LM&Ro 36 166 0 28 Sep 2023
Qwen Technical Report Jinze Bai Shuai Bai Yunfei Chu Zeyu Cui Kai Dang ... Zhenru Zhang Chang Zhou Jingren Zhou Xiaohuan Zhou Tianhang Zhu OSLM 29 1,577 0 28 Sep 2023
VDC: Versatile Data Cleanser based on Visual-Linguistic Inconsistency by Multimodal Large Language Models Daniele De Sensi Mingda Zhang Salvatore Di Girolamo Bing Wu Torsten Hoefler MLLM 30 3 0 28 Sep 2023
AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Model Avamarie Brueggeman Andrea Madotto Zhaojiang Lin Tushar Nagarajan Matt Smith ... Peyman Heidari Yue Liu Kavya Srinet Babak Damavandi Anuj Kumar MLLM 34 93 0 27 Sep 2023
BT-Adapter: Video Conversation is Feasible Without Video Instruction Tuning Ruyang Liu Chen Li Yixiao Ge Ying Shan Thomas H. Li Ge Li 25 28 0 27 Sep 2023
NLPBench: Evaluating Large Language Models on Solving NLP Problems Linxin Song Jieyu Zhang Lechao Cheng Pengyuan Zhou Tianyi Zhou Irene Z Li ELM LM&MA LRM 28 10 0 27 Sep 2023
Jointly Training Large Autoregressive Multimodal Models Emanuele Aiello L. Yu Yixin Nie Armen Aghajanyan Barlas Oğuz 19 29 0 27 Sep 2023
Rethinking Channel Dimensions to Isolate Outliers for Low-bit Weight Quantization of Large Language Models Jung Hwan Heo Jeonghoon Kim Beomseok Kwon Byeongwook Kim Se Jung Kwon Dongsoo Lee MQ 40 9 0 27 Sep 2023
InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition Pan Zhang Xiaoyi Wang Bin Wang Yuhang Cao Chao Xu ... Conghui He Xingcheng Zhang Yu Qiao Da Lin Jiaqi Wang MLLM 68 222 0 26 Sep 2023
MSG-BART: Multi-granularity Scene Graph-Enhanced Encoder-Decoder Language Model for Video-grounded Dialogue Generation Hongcheng Liu Zhe Chen Hui Li Pingjie Wang Yanfeng Wang Yu Wang VGen 43 1 0 26 Sep 2023
Aligning Large Multimodal Models with Factually Augmented RLHF Zhiqing Sun Sheng Shen Shengcao Cao Haotian Liu Chunyuan Li ... Liangyan Gui Yu-xiong Wang Yiming Yang Kurt Keutzer Trevor Darrell VLM 39 311 0 25 Sep 2023
DeepSpeed-VisualChat: Multi-Round Multi-Image Interleave Chat via Multi-Modal Causal Attention Z. Yao Xiaoxia Wu Conglong Li Minjia Zhang Heyang Qi Olatunji Ruwase A. A. Awan Samyam Rajbhandari Yuxiong He 31 11 0 25 Sep 2023
Natural Language based Context Modeling and Reasoning for Ubiquitous Computing with Large Language Models: A Tutorial Haoyi Xiong Jiang Bian Sijia Yang Xiaofei Zhang Linghe Kong Daqing Zhang LRM LLMAG 35 5 0 24 Sep 2023
A Survey on Image-text Multimodal Models Ruifeng Guo Jingxuan Wei Linzhuang Sun Khai Le-Duc Guiyong Chang Dawei Liu Sibo Zhang Zhengbing Yao Mingjun Xu Liping Bu VLM 31 5 0 23 Sep 2023