Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding

5 June 2023

Xin Li

Papers citing "Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding"

50 / 696 papers shown

Title
Empowering Biomedical Discovery with AI Agents Shanghua Gao Ada Fang Yepeng Huang Valentina Giunchiglia Ayush Noori Jonathan Richard Schwarz Yasha Ektefaie Jovana Kondic Marinka Zitnik LLMAG AI4CE 39 66 0 03 Apr 2024
MotionChain: Conversational Motion Controllers via Multimodal Prompts Biao Jiang Xin Chen C. Zhang Fukun Yin Zhuoyuan Li Gang Yu Jiayuan Fan VGen LRM 29 10 0 02 Apr 2024
CausalChaos! Dataset for Comprehensive Causal Action Question Answering Over Longer Causal Chains Grounded in Dynamic Visual Scenes Paritosh Parmar Eric Peh Ruirui Chen Ting En Lam Yuhan Chen Elston Tan Basura Fernando CML 22 7 0 01 Apr 2024
Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward Ruohong Zhang Liangke Gui Zhiqing Sun Yihao Feng Keyang Xu ... Di Fu Chunyuan Li Alexander G. Hauptmann Yonatan Bisk Yiming Yang MLLM 43 57 0 01 Apr 2024
ST-LLM: Large Language Models Are Effective Temporal Learners Ruyang Liu Chen Li Haoran Tang Yixiao Ge Ying Shan Ge Li 27 68 0 30 Mar 2024
LITA: Language Instructed Temporal-Localization Assistant De-An Huang Shijia Liao Subhashree Radhakrishnan Hongxu Yin Pavlo Molchanov Zhiding Yu Jan Kautz VLM 45 49 0 27 Mar 2024
Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models Yanwei Li Yuechen Zhang Chengyao Wang Zhisheng Zhong Yixin Chen Ruihang Chu Shaoteng Liu Jiaya Jia VLM MLLM MoE 32 211 0 27 Mar 2024
An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLM Wonkyun Kim Changin Choi Wonseok Lee Wonjong Rhee VLM 43 50 0 27 Mar 2024
Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning Hao Shao Shengju Qian Han Xiao Guanglu Song Zhuofan Zong Letian Wang Yu Liu Hongsheng Li VGen LRM MLLM 58 36 0 25 Mar 2024
Elysium: Exploring Object-level Perception in Videos via MLLM Hang Wang Yanjie Wang Yongjie Ye Yuxiang Nie Can Huang MLLM 32 19 0 25 Mar 2024
AVicuna: Audio-Visual LLM with Interleaver and Context-Boundary Alignment for Temporal Referential Dialogue Yunlong Tang Daiki Shimada Jing Bi Chenliang Xu VGen 27 10 0 24 Mar 2024
Hear Me, See Me, Understand Me: Audio-Visual Autism Behavior Recognition Shijian Deng Erin E. Kosloski Siddhi Patel Zeke A. Barnett Yiyang Nan ... William T. Doan Matthew Wang Harsh Singh P. Rollins Yapeng Tian 26 4 0 22 Mar 2024
LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models Yuzhang Shang Mu Cai Bingxin Xu Yong Jae Lee Yan Yan VLM 29 104 0 22 Mar 2024
InternVideo2: Scaling Video Foundation Models for Multimodal Video Understanding Yi Wang Kunchang Li Xinhao Li Jiashuo Yu Yinan He ... Hongjie Zhang Yifei Huang Yu Qiao Yali Wang Limin Wang 27 44 0 22 Mar 2024
MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems? Renrui Zhang Dongzhi Jiang Yichi Zhang Haokun Lin Ziyu Guo ... Aojun Zhou Pan Lu Kai-Wei Chang Peng Gao Hongsheng Li 27 167 0 21 Mar 2024
FMM-Attack: A Flow-based Multi-modal Adversarial Attack on Video-based LLMs Jinmin Li Kuofeng Gao Yang Bai Jingyun Zhang Shu-Tao Xia Yisen Wang AAML 22 7 0 20 Mar 2024
Mora: Enabling Generalist Video Generation via A Multi-Agent Framework Zhengqing Yuan Ruoxi Chen Zhaoxu Li Haolong Jia Lifang He Chi Wang Lichao Sun VGen 53 27 0 20 Mar 2024
HYDRA: A Hyper Agent for Dynamic Compositional Visual Reasoning Fucai Ke Zhixi Cai Simindokht Jahangard Weiqing Wang P. D. Haghighi Hamid Rezatofighi LRM 38 9 0 19 Mar 2024
RelationVLM: Making Large Vision-Language Models Understand Visual Relations Zhipeng Huang Zhizheng Zhang Zheng-Jun Zha Yan Lu Baining Guo VLM 36 3 0 19 Mar 2024
Contextual AD Narration with Interleaved Multimodal Sequence Hanlin Wang Zhan Tong Kecheng Zheng Yujun Shen Limin Wang VGen 47 4 0 19 Mar 2024
VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding Yue Fan Xiaojian Ma Rujie Wu Yuntao Du Jiaqi Li Zhi Gao Qing Li VLM LLMAG 46 55 0 18 Mar 2024
Towards Neuro-Symbolic Video Understanding Minkyu Choi Harsh Goel Mohammad Omama Yunhao Yang Sahil Shah Sandeep P. Chinchali 27 8 0 16 Mar 2024
HawkEye: Training Video-Text LLMs for Grounding Text in Videos Yueqian Wang Xiaojun Meng Jianxin Liang Yuxuan Wang Qun Liu Dongyan Zhao 32 30 0 15 Mar 2024
NaturalVLM: Leveraging Fine-grained Natural Language for Affordance-Guided Visual Manipulation Ran Xu Yan Shen Xiaoqi Li Ruihai Wu Hao Dong LM&Ro 30 8 0 13 Mar 2024
Knowledge Conflicts for LLMs: A Survey Rongwu Xu Zehan Qi Zhijiang Guo Cunxiang Wang Hongru Wang Yue Zhang Wei Xu 198 91 0 13 Mar 2024
DialogGen: Multi-modal Interactive Dialogue System for Multi-turn Text-to-Image Generation Minbin Huang Yanxin Long Xinchi Deng Ruihang Chu Jiangfeng Xiong Xiaodan Liang Hong Cheng Qinglin Lu Wei Liu MLLM EGVM 59 8 0 13 Mar 2024
TutoAI: A Cross-domain Framework for AI-assisted Mixed-media Tutorial Creation on Physical Tasks Yuexi Chen Vlad I. Morariu Anh Truong Zhicheng Liu DiffM VGen 21 3 0 12 Mar 2024
Amharic LLaMA and LLaVA: Multimodal LLMs for Low Resource Languages Michael Andersland 14 0 0 11 Mar 2024
CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarios Qilang Ye Zitong Yu Rui Shao Xinyu Xie Philip H. S. Torr Xiaochun Cao MLLM 33 24 0 07 Mar 2024
Multimodal Large Language Models to Support Real-World Fact-Checking Jiahui Geng Yova Kementchedjhieva Preslav Nakov Iryna Gurevych LRM 25 11 0 06 Mar 2024
Data Augmentation using Large Language Models: Data Perspectives, Learning Paradigms and Challenges Bosheng Ding Chengwei Qin Ruochen Zhao Tianze Luo Xinze Li Guizhen Chen Wenhan Xia Junjie Hu A. Luu Shafiq R. Joty 29 18 0 05 Mar 2024
MovieLLM: Enhancing Long Video Understanding with AI-Generated Movies Zhende Song Chenchen Wang Jiamu Sheng C. Zhang Gang Yu Jiayuan Fan Tao Chen VGen 25 18 0 03 Mar 2024
Evaluating Large Language Models as Virtual Annotators for Time-series Physical Sensing Data Aritra Hota S. Chatterjee Sandip Chakraborty 27 10 0 02 Mar 2024
Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers Tsai-Shien Chen Aliaksandr Siarohin Willi Menapace Ekaterina Deyneka Hsiang-wei Chao ... Yuwei Fang Hsin-Ying Lee Jian Ren Ming-Hsuan Yang Sergey Tulyakov VGen 70 177 0 29 Feb 2024
Navigating Hallucinations for Reasoning of Unintentional Activities Shresth Grover Vibhav Vineet Y. S. Rawat LRM 42 1 0 29 Feb 2024
OSCaR: Object State Captioning and State Change Representation Nguyen Nguyen Jing Bi A. Vosoughi Yapeng Tian Pooyan Fazli Chenliang Xu 40 8 0 27 Feb 2024
PhyGrasp: Generalizing Robotic Grasping with Physics-informed Large Multimodal Models Dingkun Guo Yuqi Xiang Shuqi Zhao Xinghao Zhu Masayoshi Tomizuka Mingyu Ding Wei Zhan 24 10 0 26 Feb 2024
RoboCodeX: Multimodal Code Generation for Robotic Behavior Synthesis Yao Mu Junting Chen Qinglong Zhang Shoufa Chen Qiaojun Yu ... Wenhai Wang Jifeng Dai Yu Qiao Mingyu Ding Ping Luo 40 21 0 25 Feb 2024
LSTP: Language-guided Spatial-Temporal Prompt Learning for Long-form Video-Text Understanding Yuxuan Wang Yueqian Wang Pengfei Wu Jianxin Liang Dongyan Zhao Zilong Zheng VLM 21 9 0 25 Feb 2024
Mitigating Fine-tuning based Jailbreak Attack with Backdoor Enhanced Safety Alignment Jiong Wang Jiazhao Li Yiquan Li Xiangyu Qi Junjie Hu Yixuan Li P. McDaniel Muhao Chen Bo Li Chaowei Xiao AAML SILM 32 16 0 22 Feb 2024
Enhancing Robotic Manipulation with AI Feedback from Multimodal Large Language Models Jinyi Liu Yifu Yuan Jianye Hao Fei Ni Lingzhi Fu Yibin Chen Yan Zheng LM&Ro 57 4 0 22 Feb 2024
LLMs Meet Long Video: Advancing Long Video Comprehension with An Interactive Visual Adapter in LLMs Yunxin Li Xinyu Chen Baotain Hu Min-Ling Zhang 35 3 0 21 Feb 2024
Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions Akash Ghosh Arkadeep Acharya Sriparna Saha Vinija Jain Aman Chadha VLM 41 23 0 20 Feb 2024
VideoPrism: A Foundational Visual Encoder for Video Understanding Long Zhao N. B. Gundavarapu Liangzhe Yuan Hao Zhou Shen Yan ... Huisheng Wang Hartwig Adam Mikhail Sirotenko Ting Liu Boqing Gong VGen 27 29 0 20 Feb 2024
Slot-VLM: SlowFast Slots for Video-Language Modeling Jiaqi Xu Cuiling Lan Wenxuan Xie Xuejin Chen Yan Lu MLLM VLM 35 7 0 20 Feb 2024
Model Composition for Multimodal Large Language Models Chi Chen Yiyang Du Zheng Fang Ziyue Wang Fuwen Luo ... Ming Yan Ji Zhang Fei Huang Maosong Sun Yang Janet Liu MoMe 24 3 0 20 Feb 2024
The Revolution of Multimodal Large Language Models: A Survey Davide Caffagni Federico Cocchi Luca Barsellotti Nicholas Moratelli Sara Sarto Lorenzo Baraldi Lorenzo Baraldi Marcella Cornia Rita Cucchiara LRM VLM 49 41 0 19 Feb 2024
LVCHAT: Facilitating Long Video Comprehension Yu-Xiang Wang Zeyuan Zhang Julian McAuley Zexue He VLM 26 4 0 19 Feb 2024
Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Models Didi Zhu Zhongyi Sun Zexi Li Tao Shen Ke Yan Shouhong Ding Kun Kuang Chao Wu CLL KELM MoMe 55 22 0 19 Feb 2024
CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automation Xinbei Ma Zhuosheng Zhang Hai Zhao LLMAG 33 21 0 19 Feb 2024