Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding

5 June 2023

Xin Li

Papers citing "Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding"

50 / 696 papers shown

Title
MotionBank: A Large-scale Video Motion Benchmark with Disentangled Rule-based Annotations Liang Xu Shaoyang Hua Zili Lin Yifan Liu Feipeng Ma Yichao Yan Xin Jin Xiaokang Yang Wenjun Zeng VGen 39 3 0 17 Oct 2024
Roadmap towards Superhuman Speech Understanding using Large Language Models Fan Bu Yuhao Zhang X. Wang Benyou Wang Q. Liu H. Li LM&MA ELM AuLLM 72 1 0 17 Oct 2024
The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audio Sicong Leng Yun Xing Zesen Cheng Yang Zhou Hang Zhang Xin Li Deli Zhao Shijian Lu Chunyan Miao Lidong Bing 25 7 0 16 Oct 2024
MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMs Yunqiu Xu Linchao Zhu Yi Yang 23 3 0 16 Oct 2024
TransAgent: Transfer Vision-Language Foundation Models with Heterogeneous Agent Collaboration Yiwei Guo Shaobin Zhuang Kunchang Li Yu Qiao Yali Wang VLM CLIP 21 0 0 16 Oct 2024
OMCAT: Omni Context Aware Transformer Arushi Goel Karan Sapra Matthieu Le Rafael Valle Andrew Tao Bryan Catanzaro MLLM VLM 16 0 0 15 Oct 2024
It's Just Another Day: Unique Video Captioning by Discriminative Prompting Toby Perrett Tengda Han Dima Damen Andrew Zisserman 19 3 0 15 Oct 2024
VidCompress: Memory-Enhanced Temporal Compression for Video Understanding in Large Language Models Xiaohan Lan Yitian Yuan Zequn Jie Lin Ma VLM 19 2 0 15 Oct 2024
Character-aware audio-visual subtitling in context Jaesung Huh Andrew Zisserman 31 0 0 14 Oct 2024
When Does Perceptual Alignment Benefit Vision Representations? Shobhita Sundaram Stephanie Fu Lukas Muttenthaler Netanel Y. Tamir Lucy Chai Simon Kornblith Trevor Darrell Phillip Isola 49 6 1 14 Oct 2024
Free Video-LLM: Prompt-guided Visual Perception for Efficient Training-free Video LLMs Kai Han Jianyuan Guo Yehui Tang W. He Enhua Wu Yunhe Wang MLLM VLM 21 3 0 14 Oct 2024
Surgical-LLaVA: Toward Surgical Scenario Understanding via Large Language and Vision Models Juseong Jin Chang Wook Jeong 18 3 0 13 Oct 2024
Towards Efficient Visual-Language Alignment of the Q-Former for Visual Reasoning Tasks Sungkyung Kim Adam Lee Junyoung Park Andrew Chung Jusang Oh Jay-Yoon Lee 24 3 0 12 Oct 2024
Audio Description Generation in the Era of LLMs and VLMs: A Review of Transferable Generative AI Technologies Yingqiang Gao Lukas Fischer Alexa Lintner Sarah Ebling 27 0 0 11 Oct 2024
Sample then Identify: A General Framework for Risk Control and Assessment in Multimodal Large Language Models Qingni Wang Tiantian Geng Zhiyuan Wang Teng Wang Bo Fu Feng Zheng 25 4 0 10 Oct 2024
$G$^{2}$TR: Generalized Grounded Temporal Reasoning for Robot Instruction Following by Combining Large Pre-trained Models$ G $^{2}$ TR: Generalized Grounded Temporal Reasoning for Robot Instruction Following by Combining Large Pre-trained Models Riya Arora N. N. Aman Tambi Sandeep S. Zachariah Souvik Chakraborty Rohan Paul LM&Ro 28 0 0 10 Oct 2024
Enhancing Multimodal LLM for Detailed and Accurate Video Captioning using Multi-Round Preference Optimization Changli Tang Yixuan Li Yudong Yang Jimin Zhuang Guangzhi Sun Wei Li Z. Ma Chao Zhang 25 4 0 09 Oct 2024
MM-Ego: Towards Building Egocentric Multimodal LLMs for Video QA Hanrong Ye Haotian Zhang Erik Daxberger Lin Chen Zongyu Lin ... Haoxuan You Dan Xu Zhe Gan Jiasen Lu Yinfei Yang EgoV MLLM 70 12 0 09 Oct 2024
Temporal Reasoning Transfer from Text to Video Lei Li Yuanxin Liu Linli Yao Peiyuan Zhang Chenxin An Lean Wang Xu Sun Lingpeng Kong Qi Liu LRM 37 6 0 08 Oct 2024
Grounding is All You Need? Dual Temporal Grounding for Video Dialog You Qin Wei Ji Xinze Lan Hao Fei Xun Yang Dan Guo Roger Zimmermann Lizi Liao VGen 41 0 0 08 Oct 2024
Enhancing Temporal Modeling of Video LLMs via Time Gating Zi-Yuan Hu Yiwu Zhong Shijia Huang M. Lyu Liwei Wang VLM 26 0 0 08 Oct 2024
Multimodal Large Language Models and Tunings: Vision, Language, Sensors, Audio, and Beyond Soyeon Caren Han Feiqi Cao Josiah Poon Roberto Navigli MLLM VLM 18 5 0 08 Oct 2024
TRACE: Temporal Grounding Video LLM via Causal Event Modeling Yongxin Guo Jingyu Liu Mingda Li Xiaoying Tang Qingbin Liu Xiaoying Tang 30 14 0 08 Oct 2024
Polymath: A Challenging Multi-modal Mathematical Reasoning Benchmark Himanshu Gupta Shreyas Verma Ujjwala Anantheswaran Kevin Scaria Mihir Parmar Swaroop Mishra Chitta Baral ReLM LRM 24 4 0 06 Oct 2024
Realizing Video Summarization from the Path of Language-based Semantic Understanding Kuan-Chen Mu Zhi-Yi Chin Wei-Chen Chiu 13 0 0 06 Oct 2024
Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Models Haibo Wang Zhiyang Xu Yu Cheng Shizhe Diao Yufan Zhou Yixin Cao Qifan Wang Weifeng Ge Lifu Huang 22 20 0 04 Oct 2024
StoryNavi: On-Demand Narrative-Driven Reconstruction of Video Play With Generative AI Alston Lantian Xu Tianwei Ma Tianmeng Liu Can Liu Alvaro Cassinelli VGen 29 0 0 04 Oct 2024
Self-Powered LLM Modality Expansion for Large Speech-Text Models Tengfei Yu Xuebo Liu Zhiyi Hou Liang Ding Dacheng Tao Min Zhang 32 0 0 04 Oct 2024
Frame-Voyager: Learning to Query Frames for Video Large Language Models Sicheng Yu Chengkai Jin Huanyu Wang Zhenghao Chen Sheng Jin ... Zhenbang Sun Bingni Zhang Jiawei Wu Hao Zhang Qianru Sun 67 5 0 04 Oct 2024
AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark Wenhao Chai Enxin Song Y. Du Chenlin Meng Vashisht Madhavan Omer Bar-Tal Jeng-Neng Hwang Saining Xie Christopher D. Manning 3DV 77 25 0 04 Oct 2024
Video Instruction Tuning With Synthetic Data Yuanhan Zhang Jinming Wu Wei Li Bo Li Zejun Ma Ziwei Liu Chunyuan Li SyDa VGen 39 136 0 03 Oct 2024
Visual Prompting in LLMs for Enhancing Emotion Recognition Qixuan Zhang Zhifeng Wang Dylan Zhang Wenjia Niu Sabrina Caldwell Tom Gedeon Yang Liu Zhenyue Qin 25 0 0 03 Oct 2024
Open-vocabulary Multimodal Emotion Recognition: Dataset, Metric, and Benchmark Zheng Lian Haiyang Sun Licai Sun Lan Chen Haoyu Chen ... Rui Liu Shan Liang Ya Li Jiangyan Yi Jianhua Tao VLM 25 0 0 02 Oct 2024
UAL-Bench: The First Comprehensive Unusual Activity Localization Benchmark Hasnat Md Abdullah Tian Liu Kangda Wei Shu Kong Ruihong Huang 29 3 0 02 Oct 2024
VideoCLIP-XL: Advancing Long Description Understanding for Video CLIP Models Jiapeng Wang Chengyu Wang Kunzhe Huang Jun Huang Lianwen Jin CLIP VLM 27 3 0 01 Oct 2024
Efficient Driving Behavior Narration and Reasoning on Edge Device Using Large Language Models Yizhou Huang Yihua Cheng Kezhi Wang LRM 40 1 0 30 Sep 2024
One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos Zechen Bai Tong He Haiyang Mei Pichao Wang Ziteng Gao Joya Chen Lei Liu Zheng Zhang Mike Zheng Shou VLM VOS MLLM 37 17 0 29 Sep 2024
Video DataFlywheel: Resolving the Impossible Data Trinity in Video-Language Understanding Xiao Wang Jianlong Wu Zijia Lin Fuzheng Zhang Di Zhang Liqiang Nie VGen 25 1 0 29 Sep 2024
From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understanding Heqing Zou Tianze Luo Guiyang Xie Victor Zhang ... Guangcong Wang Juanyang Chen Zhuochen Wang Hansheng Zhang Huaijian Zhang VLM 34 6 0 27 Sep 2024
EgoLM: Multi-Modal Language Model of Egocentric Motions Fangzhou Hong Vladimir Guzov Hyo Jin Kim Yuting Ye Richard A. Newcombe Ziwei Liu Lingni Ma 32 5 0 26 Sep 2024
E.T. Bench: Towards Open-Ended Event-Level Video-Language Understanding Ye Liu Zongyang Ma Zhongang Qi Yang Wu Ying Shan Chang Wen Chen 31 16 0 26 Sep 2024
LLM4Brain: Training a Large Language Model for Brain Video Understanding Ruizhe Zheng Lichao Sun 24 0 0 26 Sep 2024
MECD: Unlocking Multi-Event Causal Discovery in Video Reasoning Tieyuan Chen Huabin Liu Tianyao He Yihang Chen Chaofan Gan ... Cheng Zhong Yang Zhang Yingxue Wang Hui Lin Weiyao Lin VGen CML 35 5 0 26 Sep 2024
EAGLE: Egocentric AGgregated Language-video Engine Jing Bi Yunlong Tang Luchuan Song A. Vosoughi Nguyen Nguyen Chenliang Xu 30 8 0 26 Sep 2024
EAGLE: Towards Efficient Arbitrary Referring Visual Prompts Comprehension for Multimodal Large Language Models Jiacheng Zhang Yang Jiao Shaoxiang Chen Jingjing Chen Yu-Gang Jiang 18 1 0 25 Sep 2024
EventHallusion: Diagnosing Event Hallucinations in Video LLMs Jiacheng Zhang Yang Jiao Shaoxiang Chen Jingjing Chen Zhiyu Tan Hao Li Jingjing Chen MLLM 59 17 0 25 Sep 2024
Multi-Modal Generative AI: Multi-modal LLM, Diffusion and Beyond Hong Chen Xin Wang Yuwei Zhou Bin Huang Yipeng Zhang Wei Feng Houlun Chen Zeyang Zhang Siao Tang Wenwu Zhu DiffM 44 7 0 23 Sep 2024
Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding Yan Shu Peitian Zhang Zheng Liu Minghao Qin Junjie Zhou Tiejun Huang Bo Zhao VLM 50 41 0 22 Sep 2024
Interpolating Video-LLMs: Toward Longer-sequence LMMs in a Training-free Manner Yuzhang Shang Bingxin Xu Weitai Kang Mu Cai Yuheng Li Zehao Wen Zhen Dong Kurt Keutzer Yong Jae Lee Yan Yan 33 7 0 19 Sep 2024
MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Engines Dongzhi Jiang Renrui Zhang Ziyu Guo Yanmin Wu Jiayi Lei ... Guanglu Song Peng Gao Yu Liu Chunyuan Li Hongsheng Li MLLM 27 16 0 19 Sep 2024