VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset

29 May 2023

Papers citing "VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset"

50 / 80 papers shown

Title
RTV-Bench: Benchmarking MLLM Continuous Perception, Understanding and Reasoning through Real-Time Video Shuhang Xun Sicheng Tao J. Li Yibo Shi Zhixin Lin ... Shikang Wang Y. Liu H. Zhang Ying Ma Xuming Hu VLM LRM 41 0 0 04 May 2025
VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models Weiye Xu J. Wang Weiyun Wang Zhe Chen Wengang Zhou ... Xiaohua Wang Xizhou Zhu Wenhai Wang Jifeng Dai Jinguo Zhu VLM LRM 48 0 0 21 Apr 2025
A Survey on Cross-Modal Interaction Between Music and Multimodal Data Sifei Li Mining Tan Feier Shen Minyan Luo Zijiao Yin Fan Tang W. Dong Changsheng Xu 57 0 0 17 Apr 2025
LVC: A Lightweight Compression Framework for Enhancing VLMs in Long Video Understanding Ziyi Wang Haoran Wu Yiming Rong Deyang Jiang Yixin Zhang Y. Zhao Shuang Xu Bo Xu VLM 41 0 0 09 Apr 2025
Safety Modulation: Enhancing Safety in Reinforcement Learning through Cost-Modulated Rewards Hanping Zhang Yuhong Guo OffRL 31 0 0 03 Apr 2025
Leveraging Modality Tags for Enhanced Cross-Modal Video Retrieval A. Fragomeni Dima Damen Michael Wray 33 0 0 02 Apr 2025
WikiVideo: Article Generation from Multiple Videos Alexander Martin Reno Kriz William Walden Kate Sanders Hannah Recknor Eugene Yang Francis Ferraro Benjamin Van Durme DiffM VGen 44 1 0 01 Apr 2025
Chapter-Llama: Efficient Chaptering in Hour-Long Videos with LLMs Lucas Ventura Antoine Yang Cordelia Schmid Gül Varol 28 0 0 31 Mar 2025
Aurelia: Test-time Reasoning Distillation in Audio-Visual LLMs Sanjoy Chowdhury Hanan Gani Nishit Anand Sayan Nag Ruohan Gao Mohamed Elhoseiny Salman Khan Dinesh Manocha LRM 36 0 0 29 Mar 2025
MMMORRF: Multimodal Multilingual Modularized Reciprocal Rank Fusion Saron Samuel Dan DeGenaro Jimena Guallar-Blasco Kate Sanders Oluwaseun Eisape ... David Etter Efsun Kayi Matthew Wiesner Kenton W. Murray Reno Kriz 83 0 0 26 Mar 2025
ACVUBench: Audio-Centric Video Understanding Benchmark Y. Yang Jimin Zhuang Guangzhi Sun Changli Tang Y. Li P. Li Yifan Jiang W. Li Z. Ma Chao Zhang AuLLM CoGe 53 0 0 25 Mar 2025
PAVE: Patching and Adapting Video Large Language Models Zhuoming Liu Yiquan Li Khoi Duc Nguyen Yiwu Zhong Yin Li KELM LRM 79 0 0 25 Mar 2025
Video-ColBERT: Contextualized Late Interaction for Text-to-Video Retrieval Arun V. Reddy Alexander Martin Eugene Yang Andrew Yates Kate Sanders Kenton W. Murray Reno Kriz Celso M. De Melo Benjamin Van Durme Rama Chellappa 44 1 0 24 Mar 2025
Unbiasing through Textual Descriptions: Mitigating Representation Bias in Video Benchmarks Nina Shvetsova Arsha Nagrani Bernt Schiele Hilde Kuehne Christian Rupprecht 42 0 0 24 Mar 2025
Quantum EigenGame for excited state calculation David Quiroga Jason Han Anastasios Kyrillidis 48 0 0 17 Mar 2025
DAVE: Diagnostic benchmark for Audio Visual Evaluation Gorjan Radevski Teodora Popordanoska Matthew B. Blaschko Tinne Tuytelaars 53 0 0 12 Mar 2025
HarmonySet: A Comprehensive Dataset for Understanding Video-Music Semantic Alignment and Temporal Synchronization Zitang Zhou Ke Mei Yu Lu Tianyi Wang Fengyun Rao 83 2 0 03 Mar 2025
Pretrained Image-Text Models are Secretly Video Captioners Chunhui Zhang Yiren Jian Z. Ouyang Soroush Vosoughi VLM 63 3 0 20 Feb 2025
video-SALMONN-o1: Reasoning-enhanced Audio-visual Large Language Model Guangzhi Sun Yudong Yang Jimin Zhuang Changli Tang Y. Li W. Li Z. Ma Chao Zhang LRM MLLM VLM 64 2 0 17 Feb 2025
Learning Human Skill Generators at Key-Step Levels Yilu Wu Chenhui Zhu Shuai Wang Hanlin Wang Jing Wang Zhaoxiang Zhang Limin Wang VGen 112 0 0 12 Feb 2025
A Comprehensive Survey of Foundation Models in Medicine Wasif Khan Seowung Leem Kyle B. See Joshua K. Wong Shaoting Zhang R. Fang AI4CE LM&MA VLM 95 16 0 17 Jan 2025
OneLLM: One Framework to Align All Modalities with Language Jiaming Han Kaixiong Gong Yiyuan Zhang Jiaqi Wang Kaipeng Zhang D. Lin Yu Qiao Peng Gao Xiangyu Yue MLLM 104 102 0 10 Jan 2025
Audio-Language Datasets of Scenes and Events: A Survey Gijs Wijngaard Elia Formisano Michele Esposito M. Dumontier 76 2 0 10 Jan 2025
A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Future Shilin Sun Wenbin An Feng Tian Fang Nan Qidong Liu J. Liu N. Shah Ping Chen 78 2 0 18 Dec 2024
Do Language Models Understand Time? Xi Ding Lei Wang 162 0 0 18 Dec 2024
Gramian Multimodal Representation Learning and Alignment Giordano Cicchetti Eleonora Grassucci Luigi Sigillo Danilo Comminiello 76 0 0 16 Dec 2024
Multi-Modal interpretable automatic video captioning Antoine Hanna-Asaad Decky Aspandi Titus Zaharia 23 0 0 11 Nov 2024
Sensor2Text: Enabling Natural Language Interactions for Daily Activity Tracking Using Wearable Sensors Wenqiang Chen Jiaxuan Cheng Leyao Wang Wei Zhao Wojciech Matusik 23 1 0 26 Oct 2024
AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Models Kim Sung-Bin Oh Hyun-Bin JungMok Lee Arda Senocak Joon Son Chung Tae-Hyun Oh MLLM VLM 29 2 0 23 Oct 2024
OMCAT: Omni Context Aware Transformer Arushi Goel Karan Sapra Matthieu Le Rafael Valle Andrew Tao Bryan Catanzaro MLLM VLM 16 0 0 15 Oct 2024
LocoMotion: Learning Motion-Focused Video-Language Representations Hazel Doughty Fida Mohammad Thoker Cees G. M. Snoek 27 2 0 15 Oct 2024
MultiVENT 2.0: A Massive Multilingual Benchmark for Event-Centric Video Retrieval Reno Kriz Kate Sanders David Etter Kenton W. Murray Cameron Carpenter ... Alexander Martin Ronald Colaianni Nolan King Eugene Yang Benjamin Van Durme VGen 25 2 0 15 Oct 2024
TRACE: Temporal Grounding Video LLM via Causal Event Modeling Yongxin Guo Jingyu Liu Mingda Li Xiaoying Tang Qingbin Liu Xiaoying Tang 20 14 0 08 Oct 2024
Video DataFlywheel: Resolving the Impossible Data Trinity in Video-Language Understanding Xiao Wang Jianlong Wu Zijia Lin Fuzheng Zhang Di Zhang Liqiang Nie VGen 25 1 0 29 Sep 2024
Language-based Audio Moment Retrieval Hokuto Munakata Taichi Nishimura Shota Nakada Tatsuya Komatsu 28 1 0 24 Sep 2024
From Linguistic Giants to Sensory Maestros: A Survey on Cross-Modal Reasoning with Large Language Models Shengsheng Qian Zuyi Zhou Dizhan Xue Bing Wang Changsheng Xu LRM 34 1 0 19 Sep 2024
TempMe: Video Temporal Token Merging for Efficient Text-Video Retrieval Leqi Shen Tianxiang Hao Tao He Sicheng Zhao Pengzhang Liu Yongjun Bao Guiguang Ding Guiguang Ding 57 6 0 02 Sep 2024
Dissecting Temporal Understanding in Text-to-Audio Retrieval Andreea-Maria Oncescu João F. Henriques A. Sophia Koepke 17 2 0 01 Sep 2024
Estimated Audio-Caption Correspondences Improve Language-Based Audio Retrieval Paul Primus Florian Schmid Gerhard Widmer 29 0 0 21 Aug 2024
ExpertAF: Expert Actionable Feedback from Video Kumar Ashutosh Tushar Nagarajan Georgios Pavlakos Kris M. Kitani Kristen Grauman VGen 42 2 0 01 Aug 2024
EgoCVR: An Egocentric Benchmark for Fine-Grained Composed Video Retrieval Thomas Hummel Shyamgopal Karthik Mariana-Iuliana Georgescu Zeynep Akata EgoV 34 4 0 23 Jul 2024
MERLIN: Multimodal Embedding Refinement via LLM-based Iterative Navigation for Text-Video Retrieval-Rerank Pipeline D. Han Eunhwan Park Gisang Lee Adam Lee Nojun Kwak 32 2 0 17 Jul 2024
Meerkat: Audio-Visual Large Language Model for Grounding in Space and Time Sanjoy Chowdhury Sayan Nag Subhrajyoti Dasgupta Jun Chen Mohamed Elhoseiny Ruohan Gao Dinesh Manocha VLM MLLM 29 9 0 01 Jul 2024
Tarsier: Recipes for Training and Evaluating Large Video Description Models Jiawei Wang Liping Yuan Yuchen Zhang 29 52 0 30 Jun 2024
video-SALMONN: Speech-Enhanced Audio-Visual Large Language Models Guangzhi Sun Wenyi Yu Changli Tang Xianzhao Chen Tian Tan Wei Li Lu Lu Zejun Ma Yuxuan Wang Chao Zhang 45 22 0 22 Jun 2024
Explore the Limits of Omni-modal Pretraining at Scale Yiyuan Zhang Handong Li Jing Liu Xiangyu Yue VLM LRM 38 1 0 13 Jun 2024
2DP-2MRC: 2-Dimensional Pointer-based Machine Reading Comprehension Method for Multimodal Moment Retrieval Jiajun He T. Toda 24 0 0 10 Jun 2024
A Survey of Multimodal Large Language Model from A Data-centric Perspective Tianyi Bai Hao Liang Binwang Wan Yanran Xu Xi Li ... Ping-Chia Huang Jiulong Shan Conghui He Binhang Yuan Wentao Zhang 47 31 0 26 May 2024
From Sora What We Can See: A Survey of Text-to-Video Generation Rui Sun Yumin Zhang Tejal Shah Jiahao Sun Shuoying Zhang Wenqi Li Haoran Duan Bo Wei R. Ranjan EGVM 76 17 0 17 May 2024
SonifyAR: Context-Aware Sound Generation in Augmented Reality Xia Su Jon E. Froehlich Eunyee Koh Chang Xiao 30 3 0 11 May 2024