Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding

5 June 2023

Xin Li

Papers citing "Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding"

50 / 694 papers shown

Title
Rec-GPT4V: Multimodal Recommendation with Large Vision-Language Models Yuqing Liu Yu Wang Lichao Sun Philip S. Yu 12 6 0 13 Feb 2024
World Model on Million-Length Video And Language With Blockwise RingAttention Hao Liu Wilson Yan Matei A. Zaharia Pieter Abbeel VGen 29 57 0 13 Feb 2024
Walia-LLM: Enhancing Amharic-LLaMA by Integrating Task-Specific and Generative Datasets Israel Abebe Azime A. Tonja Tadesse Destaw Belay Mitiku Yohannes Fuge A. Wassie Eyasu Shiferaw Jada Yonas Chanie W. Sewunetie Seid Muhie Yimam 16 1 0 12 Feb 2024
Unsupervised Sign Language Translation and Generation Zhengsheng Guo Zhiwei He Wenxiang Jiao Xing Wang Rui Wang Kehai Chen Zhaopeng Tu Yong-mei Xu Min Zhang 44 0 0 12 Feb 2024
It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognition Chen Chen Ruizhe Li Yuchen Hu Sabato Marco Siniscalchi Pin-Yu Chen Ensiong Chng Chao-Han Huck Yang 24 19 0 08 Feb 2024
SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models Chris Liu Renrui Zhang Longtian Qiu Siyuan Huang Weifeng Lin ... Hao Shao Pan Lu Hongsheng Li Yu Qiao Peng Gao MLLM 126 106 0 08 Feb 2024
CREMA: Generalizable and Efficient Video-Language Reasoning via Multimodal Modular Fusion Shoubin Yu Jaehong Yoon Mohit Bansal 77 4 0 08 Feb 2024
RA-Rec: An Efficient ID Representation Alignment Framework for LLM-based Recommendation Xiaohan Yu Li Zhang Xin Zhao Yue Wang Zhongrui Ma 39 6 0 07 Feb 2024
Sentiment-enhanced Graph-based Sarcasm Explanation in Dialogue Kun Ouyang Liqiang Jing Xuemeng Song Meng Liu Yupeng Hu Liqiang Nie 89 3 0 06 Feb 2024
Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization Yang Jin Zhicheng Sun Kun Xu Kun Xu Liwei Chen ... Yuliang Liu Di Zhang Yang Song Kun Gai Yadong Mu VGen 47 42 0 05 Feb 2024
Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectives Sheng Luo Wei-Neng Chen Wanxin Tian Rui Liu Luanxuan Hou ... Ling Shao Yi Yang Bojun Gao Qun Li Guobin Wu 47 13 0 05 Feb 2024
A Survey for Foundation Models in Autonomous Driving Haoxiang Gao Yaqian Li Kaiwen Long Ming Yang Yiqing Shen VLM LRM 53 22 0 02 Feb 2024
Image Anything: Towards Reasoning-coherent and Training-free Multi-modal Image Generation Yuanhuiyi Lyu Xueye Zheng Lin Wang DiffM 26 9 0 31 Jan 2024
A Survey on Generative AI and LLM for Video Generation, Understanding, and Streaming Pengyuan Zhou Lin Wang Zhi Liu Yanbin Hao Pan Hui Sasu Tarkoma J. Kangasharju VGen 34 24 0 30 Jan 2024
Towards 3D Molecule-Text Interpretation in Language Models Sihang Li Zhiyuan Liu Yancheng Luo Xiang Wang Xiangnan He Kenji Kawaguchi Tat-Seng Chua Qi Tian AI4CE 24 42 0 25 Jan 2024
MM-LLMs: Recent Advances in MultiModal Large Language Models Duzhen Zhang Yahan Yu Jiahua Dong Chenxing Li Dan Su Chenhui Chu Dong Yu OffRL LRM 37 175 0 24 Jan 2024
Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences Xiyao Wang Yuhang Zhou Xiaoyu Liu Hongjin Lu Yuancheng Xu ... Taixi Lu Gedas Bertasius Mohit Bansal Huaxiu Yao Furong Huang LRM VLM 81 65 0 19 Jan 2024
Large Language Models are Efficient Learners of Noise-Robust Speech Recognition Yuchen Hu Chen Chen Chao-Han Huck Yang Ruizhe Li Chao Zhang Pin-Yu Chen Ensiong Chng 25 20 0 19 Jan 2024
MLLM-Tool: A Multimodal Large Language Model For Tool Agent Learning Chenyu Wang Weixin Luo Qianyu Chen Haonan Mai Jindi Guo Sixun Dong Xiaohua Xuan MLLM LLMAG 44 19 0 19 Jan 2024
Temporal Insight Enhancement: Mitigating Temporal Hallucination in Multimodal Large Language Models Li Sun Liuan Wang Jun Sun Takayuki Okatani MLLM 14 0 0 18 Jan 2024
On the Audio Hallucinations in Large Audio-Video Language Models Taichi Nishimura Shota Nakada Masayoshi Kondo VLM 25 5 0 18 Jan 2024
MMToM-QA: Multimodal Theory of Mind Question Answering Chuanyang Jin Yutong Wu Jing Cao Jiannan Xiang Yen-Ling Kuo Zhiting Hu T. Ullman Antonio Torralba Joshua B. Tenenbaum Tianmin Shu 25 32 0 16 Jan 2024
Towards A Better Metric for Text-to-Video Generation Jay Zhangjie Wu Guian Fang Haoning Wu Xintao Wang Yixiao Ge ... Rui Zhao Weisi Lin Wynne Hsu Ying Shan Mike Zheng Shou VGen 22 34 0 15 Jan 2024
GroundingGPT:Language Enhanced Multi-modal Grounding Model Zhaowei Li Qi Xu Dong Zhang Hang Song Yiqing Cai ... Junting Pan Zefeng Li Van Tu Vu Zhida Huang Tao Wang 18 36 0 11 Jan 2024
Video Anomaly Detection and Explanation via Large Language Models Hui Lv Qianru Sun 13 19 0 11 Jan 2024
SonicVisionLM: Playing Sound with Vision Language Models Zhifeng Xie Shengye Yu Qile He Mengtian Li VLM VGen 25 2 0 09 Jan 2024
STAIR: Spatial-Temporal Reasoning with Auditable Intermediate Results for Video Question Answering Yueqian Wang Yuxuan Wang Kai Chen Dongyan Zhao 25 2 0 08 Jan 2024
LightHouse: A Survey of AGI Hallucination Feng Wang LRM HILM VLM 24 3 0 08 Jan 2024
Towards Truly Zero-shot Compositional Visual Reasoning with LLMs as Programmers Aleksandar Stanić Sergi Caelles Michael Tschannen LRM VLM 23 9 0 03 Jan 2024
Detours for Navigating Instructional Videos Kumar Ashutosh Zihui Xue Tushar Nagarajan Kristen Grauman 16 6 0 03 Jan 2024
Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Models Xinpeng Ding Jinahua Han Hang Xu Xiaodan Liang Wei Zhang Xiaomeng Li 18 38 0 02 Jan 2024
E-chat: Emotion-sensitive Spoken Dialogue System with Large Language Models Hongfei Xue Yuhao Liang Bingshen Mu Shiliang Zhang Mengzhe Chen Qian Chen Lei Xie AuLLM 19 9 0 31 Dec 2023
Boosting Large Language Model for Speech Synthesis: An Empirical Study Hong-ping Hao Long Zhou Shujie Liu Jinyu Li Shujie Hu Rui Wang Furu Wei 29 18 0 30 Dec 2023
Video Understanding with Large Language Models: A Survey Yunlong Tang Jing Bi Siting Xu Luchuan Song Susan Liang ... Feng Zheng Jianguo Zhang Ping Luo Jiebo Luo Chenliang Xu VLM 50 81 0 29 Dec 2023
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action Jiasen Lu Christopher Clark Sangho Lee Zichen Zhang Savya Khosla Ryan Marten Derek Hoiem Aniruddha Kembhavi VLM MLLM 27 144 0 28 Dec 2023
Grounding-Prompter: Prompting LLM with Multimodal Information for Temporal Sentence Grounding in Long Videos Houlun Chen Xin Wang Hong Chen Zihan Song Jia Jia Wenwu Zhu LRM 25 10 0 28 Dec 2023
Visual Instruction Tuning towards General-Purpose Multimodal Model: A Survey Jiaxing Huang Jingyi Zhang Kai Jiang Han Qiu Shijian Lu 28 22 0 27 Dec 2023
InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks Zhe Chen Jiannan Wu Wenhai Wang Weijie Su Guo Chen ... Bin Li Ping Luo Tong Lu Yu Qiao Jifeng Dai VLM MLLM 156 895 0 21 Dec 2023
LLM4VG: Large Language Models Evaluation for Video Grounding Wei Feng Xin Wang Hong Chen Zeyang Zhang Zihan Song Yuwei Zhou Wenwu Zhu 31 8 0 21 Dec 2023
Perception Test 2023: A Summary of the First Challenge And Outcome Joseph Heyward João Carreira Dima Damen Andrew Zisserman Viorica Patraucean 14 0 0 20 Dec 2023
Text-Conditioned Resampler For Long Form Video Understanding Bruno Korbar Yongqin Xian A. Tonioni Andrew Zisserman Federico Tombari 28 12 0 19 Dec 2023
SECap: Speech Emotion Captioning with Large Language Model Yaoxun Xu Hangting Chen Jianwei Yu Qiaochu Huang Zhiyong Wu Shixiong Zhang Guangzhi Li Yi Luo Rongzhi Gu 14 22 0 16 Dec 2023
SMILE: Multimodal Dataset for Understanding Laughter in Video with Language Models Lee Hyun Kim Sung-Bin Seungju Han Youngjae Yu Tae-Hyun Oh 25 13 0 15 Dec 2023
DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving Wenhai Wang Jiangwei Xie ChuanYang Hu Haoming Zou Jianan Fan ... Lewei Lu Xizhou Zhu Xiaogang Wang Yu Qiao Jifeng Dai 34 122 0 14 Dec 2023
Assessing GPT4-V on Structured Reasoning Tasks Mukul Singh J. Cambronero Sumit Gulwani Vu Le Gust Verbruggen LRM 35 10 0 13 Dec 2023
Vista-LLaMA: Reducing Hallucination in Video Language Models via Equal Distance to Visual Tokens Fan Ma Xiaojie Jin Heng Wang Yuchen Xian Jiashi Feng Yi Yang 15 47 0 12 Dec 2023
EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planning Yi Chen Yuying Ge Yixiao Ge Mingyu Ding Bohao Li Rui Wang Rui-Lan Xu Ying Shan Xihui Liu LLMAG ELM LRM 19 9 0 11 Dec 2023
Audio-Visual LLM for Video Understanding Fangxun Shu Lei Zhang Hao Jiang Cihang Xie VLM MLLM 17 36 0 11 Dec 2023
LEGO: Learning EGOcentric Action Frame Generation via Visual Instruction Tuning Bolin Lai Xiaoliang Dai Lawrence Chen Guan Pang James M. Rehg Miao Liu 33 14 0 06 Dec 2023
Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Driving Ming-Jun Nie Renyuan Peng Chunwei Wang Xinyue Cai Jianhua Han Hang Xu Li Zhang LRM 27 44 0 06 Dec 2023