Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding

5 June 2023

Xin Li

Papers citing "Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding"

50 / 694 papers shown

Title
Circumventing shortcuts in audio-visual deepfake detection datasets with unsupervised learning Dragos-Alexandru Boldisor Stefan Smeu Dan Oneaţă Elisabeta Oneata 92 1 0 29 Nov 2024
TimeMarker: A Versatile Video-LLM for Long and Short Video Understanding with Superior Temporal Localization Ability Shimin Chen Xiaohan Lan Yitian Yuan Zequn Jie Lin Ma VLM MLLM 71 13 0 27 Nov 2024
MotionLLaMA: A Unified Framework for Motion Synthesis and Comprehension Zeyu Ling Bo Han Shiyang Li H. Shen Jikang Cheng Changqing Zou 81 1 0 26 Nov 2024
Leveraging the Power of MLLMs for Gloss-Free Sign Language Translation Jungeun Kim Hyeongwoo Jeon Jongseong Bae Ha Young Kim SLR 75 0 0 25 Nov 2024
VideoOrion: Tokenizing Object Dynamics in Videos Yicheng Feng Yijiang Li Wanpeng Zhang Sipeng Zheng Zongqing Lu Sipeng Zheng Zongqing Lu 101 1 0 25 Nov 2024
freePruner: A Training-free Approach for Large Multimodal Model Acceleration Bingxin Xu Yuzhang Shang Yunhao Ge Qian Lou Yan Yan 94 3 0 23 Nov 2024
ReWind: Understanding Long Videos with Instructed Learnable Memory Anxhelo Diko Tinghuai Wang Wassim Swaileh Shiyan Sun Ioannis Patras KELM VLM 77 0 0 23 Nov 2024
Neuro-Symbolic Evaluation of Text-to-Video Models using Formal Verification Sundar Sripada V. S. Minkyu Choi Sahil Shah Harsh Goel Mohammad Omama Sandeep P. Chinchali EGVM 108 2 0 22 Nov 2024
Beyond Training: Dynamic Token Merging for Zero-Shot Video Understanding Yiming Zhang Zhuokai Zhao Zhaorun Chen Zenghui Ding Xianjun Yang Yining Sun 122 1 0 21 Nov 2024
Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension Yongdong Luo Xiawu Zheng Xiao Yang Guilin Li Haojia Lin Jinfa Huang Jiayi Ji Fei Chao Jiebo Luo Rongrong Ji VLM 79 17 0 20 Nov 2024
On the Consistency of Video Large Language Models in Temporal Comprehension Minjoon Jung Junbin Xiao Byoung-Tak Zhang Angela Yao 83 2 0 20 Nov 2024
Generative Timelines for Instructed Visual Assembly Alejandro Pardo Jui-hsien Wang Bernard Ghanem Josef Sivic Bryan C. Russell Fabian Caba Heilbron VGen 64 0 0 19 Nov 2024
On-Board Vision-Language Models for Personalized Autonomous Vehicle Motion Control: System Design and Real-World Validation Can Cui Zichong Yang Yupeng Zhou Juntong Peng Sung-Yeon Park ... Yiheng Feng Jitesh Panchal Lingxi Li Yaobin Chen Ziran Wang 67 7 0 17 Nov 2024
TS-LLaVA: Constructing Visual Tokens through Thumbnail-and-Sampling for Training-Free Video Large Language Models Tingyu Qu Mingxiao Li Tinne Tuytelaars Marie-Francine Moens VLM 34 1 0 17 Nov 2024
DiMoDif: Discourse Modality-information Differentiation for Audio-visual Deepfake Detection and Localization C. Koutlis Symeon Papadopoulos 58 2 0 15 Nov 2024
Jailbreak Attacks and Defenses against Multimodal Generative Models: A Survey Xuannan Liu Xing Cui Peipei Li Zekun Li Huaibo Huang Shuhan Xia Miaoxuan Zhang Yueying Zou Ran He AAML 60 6 0 14 Nov 2024
VCBench: A Controllable Benchmark for Symbolic and Abstract Challenges in Video Cognition Chenglin Li Qianglong Chen Zhi Li Feng Tao Yin Zhang 29 0 0 14 Nov 2024
Spider: Any-to-Many Multimodal LLM Jinxiang Lai Jie Zhang Jun Liu Jian Li Xiaocheng Lu Song Guo MLLM 54 2 0 14 Nov 2024
Multimodal Instruction Tuning with Hybrid State Space Models Jianing Zhou Han Li Shuai Zhang Ning Xie Ruijie Wang Xiaohan Nie Sheng Liu Lingyun Wang 33 0 0 13 Nov 2024
Can MLLMs Guide Weakly-Supervised Temporal Action Localization Tasks? Quan Zhang Yuxin Qi 34 2 0 13 Nov 2024
Which Viewpoint Shows it Best? Language for Weakly Supervising View Selection in Multi-view Instructional Videos Sagnik Majumder Tushar Nagarajan Ziad Al-Halah Reina Pradhan Kristen Grauman 26 1 0 13 Nov 2024
New Emerged Security and Privacy of Pre-trained Model: a Survey and Outlook Meng Yang Tianqing Zhu Chi Liu Wanlei Zhou Shui Yu Philip S. Yu AAML ELM PILM 56 1 0 12 Nov 2024
EVQAScore: A Fine-grained Metric for Video Question Answering Data Quality Evaluation Hao Liang Zirong Chen W. Zhang Wentao Zhang 31 0 0 11 Nov 2024
HourVideo: 1-Hour Video-Language Understanding Keshigeyan Chandrasegaran Agrim Gupta Lea M. Hadzic Taran Kota Jimming He Cristobal Eyzaguirre Zane Durante Manling Li Jiajun Wu L. Fei-Fei VLM 36 31 0 07 Nov 2024
TAP-VL: Text Layout-Aware Pre-training for Enriched Vision-Language Models Jonathan Fhima Elad Ben Avraham Oren Nuriel Yair Kittenplon Roy Ganz Aviad Aberdam Ron Litman VLM 26 1 0 07 Nov 2024
VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in Videos Shehan Munasinghe Hanan Gani Wenqi Zhu Jiale Cao Eric P. Xing F. Khan Salman Khan MLLM VGen VLM 42 6 0 07 Nov 2024
MME-Finance: A Multimodal Finance Benchmark for Expert-level Understanding and Reasoning Ziliang Gan Yu Lu D. Zhang Haohan Li Che Liu ... Haipang Wu Chaoyou Fu Z. Xu Rongjunchen Zhang Yong Dai 47 3 0 05 Nov 2024
PPLLaVA: Varied Video Sequence Understanding With Prompt Guidance Ruyang Liu Haoran Tang Haibo Liu Yixiao Ge Ying Shan Chen Li Jiankun Yang VLM 40 5 0 04 Nov 2024
Contrasting with Symile: Simple Model-Agnostic Representation Learning for Unlimited Modalities A. Saporta A. Puli Mark Goldstein Rajesh Ranganath SSL 21 0 0 01 Nov 2024
Generative Emotion Cause Explanation in Multimodal Conversations Lin Wang Xiaocui Yang Shi Feng Daling Wang Yifei Zhang 34 0 0 01 Nov 2024
On Learning Multi-Modal Forgery Representation for Diffusion Generated Video Detection Xiufeng Song Xiao Guo J. Zhang Qirui Li Lei Bai Xiaoming Liu Guangtao Zhai Xiaohong Liu DiffM VGen 69 8 0 31 Oct 2024
SpeechQE: Estimating the Quality of Direct Speech Translation HyoJung Han Kevin Duh Marine Carpuat 29 0 0 28 Oct 2024
CT2C-QA: Multimodal Question Answering over Chinese Text, Table and Chart Bowen Zhao Tianhao Cheng Yuejie Zhang Ying Cheng Rui Feng Xiaobo Zhang LMTD 27 1 0 28 Oct 2024
Sensor2Text: Enabling Natural Language Interactions for Daily Activity Tracking Using Wearable Sensors Wenqiang Chen Jiaxuan Cheng Leyao Wang Wei Zhao Wojciech Matusik 26 1 0 26 Oct 2024
GiVE: Guiding Visual Encoder to Perceive Overlooked Information Junjie Li Jianghong Ma Xiaofeng Zhang Yuhang Li Jianyang Shi 23 0 0 26 Oct 2024
FLAASH: Flow-Attention Adaptive Semantic Hierarchical Fusion for Multi-Modal Tobacco Content Analysis N. V. R. Chappa P. Dobbs Bhiksha Raj Khoa Luu 32 3 0 25 Oct 2024
TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuning Xiangyu Zeng Kunchang Li Chenting Wang Xinhao Li Tianxiang Jiang ... Zhengrong Yue Yi Wang Yali Wang Yu Qiao Limin Wang MLLM VLM AI4TS 64 14 0 25 Oct 2024
MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark S. Sakshi Utkarsh Tyagi Sonal Kumar Ashish Seth Ramaneswaran Selvakumar Oriol Nieto R. Duraiswami Sreyan Ghosh Dinesh Manocha AuLLM ELM 65 22 0 24 Oct 2024
AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Models Kim Sung-Bin Oh Hyun-Bin JungMok Lee Arda Senocak Joon Son Chung Tae-Hyun Oh MLLM VLM 36 3 0 23 Oct 2024
Order Matters: Exploring Order Sensitivity in Multimodal Large Language Models Zhijie Tan Xu Chu Weiping Li Tong Mo 21 1 0 22 Oct 2024
PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction Long Xing Qidong Huang Xiaoyi Dong Jiajie Lu Pan Zhang ... Yuhang Cao Conghui He Jiaqi Wang Feng Wu Dahua Lin VLM 45 26 0 22 Oct 2024
xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video Even in VLMs Michael S Ryoo Honglu Zhou Shrikant B. Kendre Can Qin Le Xue Manli Shu Silvio Savarese R. Xu Caiming Xiong Juan Carlos Niebles VGen 34 12 0 21 Oct 2024
Mitigating Object Hallucination via Concentric Causal Attention Yun Xing Yiheng Li Ivan Laptev Shijian Lu 40 18 0 21 Oct 2024
OpenMU: Your Swiss Army Knife for Music Understanding Mengjie Zhao Zhi-Wei Zhong Zhuoyuan Mao Shiqi Yang Wei-Hsiang Liao Shusuke Takahashi Hiromi Wakaki Yuki Mitsufuji OSLM 45 4 0 21 Oct 2024
Can LVLMs Describe Videos like Humans? A Five-in-One Video Annotations Benchmark for Better Human-Machine Comparison Shiyu Hu Xuchen Li X. Li Jing Zhang Yipei Wang Xin Zhao Kang Hao Cheong VLM 26 1 0 20 Oct 2024
Making Every Frame Matter: Continuous Activity Recognition in Streaming Video via Adaptive Video Context Modeling Hao Wu Donglin Bai Shiqi Jiang Qianxi Zhang Y. Yang Ting Cao Fengyuan Xu Yunxin Liu Fengyuan Xu 60 0 0 19 Oct 2024
Assistive AI for Augmenting Human Decision-making Natabara Máté Gyöngyössy Bernát Török Csilla Farkas Laura Lucaj Attila Menyhárd Krisztina Menyhárd-Balázs András Simonyi Patrick van der Smagt Zsolt Ződi András Lőrincz 29 0 0 18 Oct 2024
Addressing Blind Guessing: Calibration of Selection Bias in Multiple-Choice Question Answering by Video Language Models Olga Loginova Oleksandr Bezrukov Alexey Kravets 18 0 0 18 Oct 2024
MotionBank: A Large-scale Video Motion Benchmark with Disentangled Rule-based Annotations Liang Xu Shaoyang Hua Zili Lin Yifan Liu Feipeng Ma Yichao Yan Xin Jin Xiaokang Yang Wenjun Zeng VGen 39 3 0 17 Oct 2024
Roadmap towards Superhuman Speech Understanding using Large Language Models Fan Bu Yuhao Zhang X. Wang Benyou Wang Q. Liu H. Li LM&MA ELM AuLLM 60 1 0 17 Oct 2024