Tarsier: Recipes for Training and Evaluating Large Video Description Models

30 June 2024

Papers citing "Tarsier: Recipes for Training and Evaluating Large Video Description Models"

44 / 44 papers shown

Title
An LLM-Empowered Low-Resolution Vision System for On-Device Human Behavior Understanding Siyang Jiang Bufang Yang Lilin Xu Mu Yuan Yeerzhati Abudunuer ... Liekang Zeng Hongkai Chen Zhenyu Yan Xiaofan Jiang Guoliang Xing VLM 37 0 0 03 May 2025
FSBench: A Figure Skating Benchmark for Advancing Artistic Sports Understanding Rong Gao Xin Liu Zhuozhao Hu Bohao Xing Baiqiang Xia Zitong Yu H. Kalviainen 38 0 0 28 Apr 2025
ActionArt: Advancing Multimodal Large Models for Fine-Grained Human-Centric Video Understanding Yi-Xing Peng Q. Yang Yu-Ming Tang Shenghao Fu Kun-Yu Lin Xihan Wei Wei-Shi Zheng 40 0 0 25 Apr 2025
VideoMultiAgents: A Multi-Agent Framework for Video Question Answering Noriyuki Kugo Xiang Li Z. Li Ashish Gupta Arpandeep Khatua ... Yuta Kyuragi Yasunori Ishii Masamoto Tanabiki Kazuki Kozuka Ehsan Adeli 49 0 0 25 Apr 2025
FRAG: Frame Selection Augmented Generation for Long Video and Long Document Understanding De-An Huang Subhashree Radhakrishnan Zhiding Yu Jan Kautz VGen VLM 71 0 0 24 Apr 2025
Towards Understanding Camera Motions in Any Video Zhiqiu Lin Siyuan Cen Daniel Jiang Jay Karhade Hewei Wang ... Rushikesh Zawar Xue Bai Yilun Du Chuang Gan Deva Ramanan VGen 21 0 0 21 Apr 2025
VideoPASTA: 7K Preference Pairs That Matter for Video-LLM Alignment Yogesh Kulkarni Pooyan Fazli 32 0 0 18 Apr 2025
Mavors: Multi-granularity Video Representation for Multimodal Large Language Model Yang Shi Jiaheng Liu Yushuo Guan Z. Wu Y. Zhang ... Bohan Zeng W. Zhang Fuzheng Zhang Wenjing Yang Di Zhang VGen VLM 63 0 0 14 Apr 2025
Capybara-OMNI: An Efficient Paradigm for Building Omni-Modal Language Models Xingguang Ji Jiakang Wang Hongzhi Zhang Jingyuan Zhang Haonan Zhou Chenxi Sun Y. Liu Qi Wang Fuzheng Zhang MLLM VLM 58 0 0 10 Apr 2025
VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuning Xinhao Li Ziang Yan Desen Meng Lu Dong Xiangyu Zeng Yinan He Y. Wang Yu Qiao Yi Wang Limin Wang VLM AI4TS LRM 34 2 0 09 Apr 2025
SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understanding Mingze Xu Mingfei Gao Shiyu Li Jiasen Lu Zhe Gan Zhengfeng Lai Meng Cao Kai Kang Y. Yang Afshin Dehghan 51 1 0 24 Mar 2025
Plug-and-Play 1.x-Bit KV Cache Quantization for Video Large Language Models Keda Tao Haoxuan You Yang Sui Can Qin H. Wang VLM MQ 81 0 0 20 Mar 2025
FAVOR-Bench: A Comprehensive Benchmark for Fine-Grained Video Motion Understanding Chongjun Tu Lin Zhang Pengtao Chen Peng Ye Xianfang Zeng W. Cheng Gang Yu Tao Chen 79 0 0 19 Mar 2025
ViSpeak: Visual Instruction Feedback in Streaming Videos Shenghao Fu Q. Yang Yuan-Ming Li Yi-Xing Peng Kun-Yu Lin Xihan Wei Jian-Fang Hu Xiaohua Xie Wei-Shi Zheng VLM 58 1 0 17 Mar 2025
Quantum EigenGame for excited state calculation David Quiroga Jason Han Anastasios Kyrillidis 48 1 0 17 Mar 2025
Vamba: Understanding Hour-Long Videos with Hybrid Mamba-Transformers Weiming Ren Wentao Ma Huan Yang Cong Wei Ge Zhang Wenhu Chen Mamba 55 3 0 14 Mar 2025
BIMBA: Selective-Scan Compression for Long-Range Video Question Answering Md. Mohaiminul Islam Tushar Nagarajan Huiyu Wang Gedas Bertasius Lorenzo Torresani 50 0 0 12 Mar 2025
VRMDiff: Text-Guided Video Referring Matting Generation of Diffusion Lehan Yang Jincen Song Tianlong Wang Daiqing Qi Weili Shi Yuheng Liu Sheng Li DiffM VOS VGen 69 0 0 11 Mar 2025
WISA: World Simulator Assistant for Physics-Aware Text-to-Video Generation Jing Wang Ao Ma Ke Cao Jun Zheng Zhanjie Zhang ... Yuhang Ma Bo Cheng Dawei Leng Yuhui Yin Xiaodan Liang VGen 79 3 0 11 Mar 2025
MEDA: Dynamic KV Cache Allocation for Efficient Multimodal Long-Context Inference Zhongwei Wan H. Shen Xin Wang C. Liu Zheda Mai M. Zhang VLM 52 3 0 24 Feb 2025
What Is a Good Caption? A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness Zhihang Liu Chen-Wei Xie Bin Wen Feiwu Yu Jixuan Chen ... Pandeng Li Yun Zheng Hongtao Xie Yun Zheng Hongtao Xie VLM CoGe 96 0 0 19 Feb 2025
Unhackable Temporal Rewarding for Scalable Video MLLMs En Yu Kangheng Lin Liang Zhao Yana Wei Zining Zhu ... Jianjian Sun Zheng Ge X. Zhang Jingyu Wang Wenbing Tao 52 4 0 17 Feb 2025
BounTCHA: A CAPTCHA Utilizing Boundary Identification in Guided Generative AI-extended Videos Lehao Lin Ke Wang Maha Abdallah Wei Cai AAML 82 0 0 30 Jan 2025
Progress-Aware Video Frame Captioning Zihui Xue Joungbin An Xitong Yang Kristen Grauman 90 1 0 03 Dec 2024
VideoSAVi: Self-Aligned Video Language Models without Human Supervision Yogesh Kulkarni Pooyan Fazli VLM 86 2 0 01 Dec 2024
PhyT2V: LLM-Guided Iterative Self-Refinement for Physics-Grounded Text-to-Video Generation Qiyao Xue Xiangyu Yin Boyuan Yang Wei Gao DiffM VGen 72 9 0 30 Nov 2024
Open-Sora Plan: Open-Source Large Video Generation Model Bin Lin Yunyang Ge Xinhua Cheng Zongjian Li Bin Zhu ... Zhang Pan Xing Zhou Shaoling Dong Yonghong Tian Li-xin Yuan VLM VGen 113 58 0 28 Nov 2024
StoryTeller: Improving Long Video Description through Global Audio-Visual Character Identification Yichen He Yuan Lin Jianchao Wu Hanchong Zhang Yuchen Zhang Ruicheng Le VGen VLM 39 2 0 11 Nov 2024
HourVideo: 1-Hour Video-Language Understanding Keshigeyan Chandrasegaran Agrim Gupta Lea M. Hadzic Taran Kota Jimming He Cristobal Eyzaguirre Zane Durante Manling Li Jiajun Wu L. Fei-Fei VLM 26 31 0 07 Nov 2024
xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video Even in VLMs Michael S Ryoo Honglu Zhou Shrikant B. Kendre Can Qin Le Xue Manli Shu Silvio Savarese R. Xu Caiming Xiong Juan Carlos Niebles VGen 24 12 0 21 Oct 2024
Can LVLMs Describe Videos like Humans? A Five-in-One Video Annotations Benchmark for Better Human-Machine Comparison Shiyu Hu Xuchen Li X. Li Jing Zhang Yipei Wang Xin Zhao Kang Hao Cheong VLM 17 1 0 20 Oct 2024
SPORTU: A Comprehensive Sports Understanding Benchmark for Multimodal Large Language Models H. Xia Zhengbang Yang Junbo Zou Rhys Tracy Yuqing Wang ... Xun Shao Zhuoqing Xie Yuan-fang Wang Weining Shen Hanjie Chen ReLM LRM ELM 28 2 0 11 Oct 2024
Video Instruction Tuning With Synthetic Data Yuanhan Zhang Jinming Wu Wei Li Bo Li Zejun Ma Ziwei Liu Chunyuan Li SyDa VGen 34 1 0 03 Oct 2024
Too Many Frames, Not All Useful: Efficient Strategies for Long-Form Video QA Jongwoo Park Kanchana Ranasinghe Kumara Kahatapitiya Wonjeong Ryoo Donghyun Kim Michael S. Ryoo 44 20 0 13 Jun 2024
VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos Ziyang Wang Shoubin Yu Elias Stengel-Eskin Jaehong Yoon Feng Cheng Gedas Bertasius Mohit Bansal 40 56 0 29 May 2024
An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLM Wonkyun Kim Changin Choi Wonseok Lee Wonjong Rhee VLM 40 50 0 27 Mar 2024
VideoAgent: Long-form Video Understanding with Large Language Model as Agent Xiaohan Wang Yuhui Zhang Orr Zohar Serena Yeung-Levy VLM 93 51 0 15 Mar 2024
Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers Tsai-Shien Chen Aliaksandr Siarohin Willi Menapace Ekaterina Deyneka Hsiang-wei Chao ... Yuwei Fang Hsin-Ying Lee Jian Ren Ming-Hsuan Yang Sergey Tulyakov VGen 67 177 0 29 Feb 2024
Video Understanding with Large Language Models: A Survey Yunlong Tang Jing Bi Siting Xu Luchuan Song Susan Liang ... Feng Zheng Jianguo Zhang Ping Luo Jiebo Luo Chenliang Xu VLM 44 76 0 29 Dec 2023
A Simple LLM Framework for Long-Range Video Question-Answering Ce Zhang Taixi Lu Md. Mohaiminul Islam Ziyang Wang Shoubin Yu Mohit Bansal Gedas Bertasius 95 80 0 28 Dec 2023
Video-LLaVA: Learning United Visual Representation by Alignment Before Projection Bin Lin Yang Ye Bin Zhu Jiaxi Cui Munan Ning Peng Jin Li-ming Yuan VLM MLLM 182 576 0 16 Nov 2023
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models Junnan Li Dongxu Li Silvio Savarese Steven C. H. Hoi VLM MLLM 244 4,186 0 30 Jan 2023
Ego4D: Around the World in 3,000 Hours of Egocentric Video Kristen Grauman Andrew Westbury Eugene Byrne Zachary Chavis Antonino Furnari ... Mike Zheng Shou Antonio Torralba Lorenzo Torresani Mingfei Yan Jitendra Malik EgoV 212 682 0 13 Oct 2021
VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding Hu Xu Gargi Ghosh Po-Yao (Bernie) Huang Dmytro Okhonko Armen Aghajanyan Florian Metze Luke Zettlemoyer Florian Metze Luke Zettlemoyer Christoph Feichtenhofer CLIP VLM 239 554 0 28 Sep 2021