MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound

7 January 2022

Yejin Choi

Papers citing "MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound"

50 / 163 papers shown

Title
TEMPURA: Temporal Event Masked Prediction and Understanding for Reasoning in Action Jen-Hao Cheng Vivian Wang Huayu Wang Huapeng Zhou Yi-Hao Peng ... Wenhao Chai Yi-Ling Chen Vibhav Vineet Qin Cai Jenq-Neng Hwang AI4TS 47 0 0 02 May 2025
Learning Streaming Video Representation via Multitask Training Yibin Yan Jilan Xu Shangzhe Di Yikun Liu Yudi Shi Qirui Chen Zeqian Li Yifei Huang Weidi Xie CLL 76 0 0 28 Apr 2025
Symbolic Representation for Any-to-Any Generative Tasks J. Chen Xiaoye Zhu Y. Wang Tianyang Liu Xinhui Chen ... Yifei Ke J. Liu Yiwen Yuan Julian McAuley Li Li DiffM 36 0 0 24 Apr 2025
REVEAL: Relation-based Video Representation Learning for Video-Question-Answering Sofian Chaybouti Walid Bousselham Moritz Wolter Hilde Kuehne 53 0 0 07 Apr 2025
Aurelia: Test-time Reasoning Distillation in Audio-Visual LLMs Sanjoy Chowdhury Hanan Gani Nishit Anand Sayan Nag Ruohan Gao Mohamed Elhoseiny Salman Khan Dinesh Manocha LRM 40 0 0 29 Mar 2025
Watch and Learn: Leveraging Expert Knowledge and Language for Surgical Video Understanding David Gastager Ghazal Ghazaei Constantin Patsch 56 0 0 14 Mar 2025
Generative Frame Sampler for Long Video Understanding Linli Yao Haoning Wu Kun Ouyang Y. Zhang Caiming Xiong Bei Chen Xu Sun Junnan Li VLM VGen 52 0 0 12 Mar 2025
SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding Zhenyu Yang Y. Hu Zemin Du Dizhan Xue Shengsheng Qian Jiahong Wu Fan Yang W. Dong Changsheng Xu 47 2 0 15 Feb 2025
2.5 Years in Class: A Multimodal Textbook for Vision-Language Pretraining Wenqi Zhang Hang Zhang Xin Li Jiashuo Sun Yongliang Shen Weiming Lu Deli Zhao Yueting Zhuang Lidong Bing VLM 37 2 0 01 Jan 2025
Neptune: The Long Orbit to Benchmarking Long Video Understanding Arsha Nagrani Mingda Zhang Ramin Mehran Rachel Hornung N. B. Gundavarapu ... Boqing Gong Cordelia Schmid Mikhail Sirotenko Yukun Zhu Tobias Weyand 100 4 0 12 Dec 2024
Video LLMs for Temporal Reasoning in Long Videos Fawad Javed Fateh Umer Ahmed Hamza Khan M. Zia Quoc-Huy Tran VLM 81 0 0 04 Dec 2024
The Sound of Water: Inferring Physical Properties from Pouring Liquids Piyush Bagad Makarand Tapaswi Cees G. M. Snoek Andrew Zisserman 40 0 0 18 Nov 2024
SPECTRUM: Semantic Processing and Emotion-informed video-Captioning Through Retrieval and Understanding Modalities Ehsan Faghihi Mohammedreza Zarenejad Ali-Asghar Beheshti Shirazi 37 0 0 04 Nov 2024
TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuning Xiangyu Zeng Kunchang Li Chenting Wang Xinhao Li Tianxiang Jiang ... Zhengrong Yue Yi Wang Yali Wang Yu Qiao Limin Wang MLLM VLM AI4TS 64 14 0 25 Oct 2024
Beyond Coarse-Grained Matching in Video-Text Retrieval Aozhu Chen Hazel Doughty Xirong Li Cees G. M. Snoek 21 0 0 16 Oct 2024
Exploring Efficient Foundational Multi-modal Models for Video Summarization Karan Samel Apoorva Beedu Nitish Sontakke Irfan Essa 20 1 0 09 Oct 2024
Video DataFlywheel: Resolving the Impossible Data Trinity in Video-Language Understanding Xiao Wang Jianlong Wu Zijia Lin Fuzheng Zhang Di Zhang Liqiang Nie VGen 25 1 0 29 Sep 2024
Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understanding Yunze Man Shuhong Zheng Zhipeng Bao M. Hebert Liang-Yan Gui Yu-xiong Wang 70 15 0 05 Sep 2024
Assessing Modality Bias in Video Question Answering Benchmarks with Multimodal Large Language Models Jean Park Kuk Jin Jang Basam Alasaly Sriharsha Mopidevi Andrew Zolensky Eric Eaton Insup Lee Kevin Johnson 26 4 0 22 Aug 2024
DIVE: Towards Descriptive and Diverse Visual Commonsense Generation Jun-Hyung Park Hyuntae Park Youjin Kang Eojin Jeon SangKeun Lee 16 0 0 15 Aug 2024
COM Kitchens: An Unedited Overhead-view Video Dataset as a Vision-Language Benchmark Koki Maeda Tosho Hirasawa Atsushi Hashimoto Jun Harashima Leszek Rybicki Yusuke Fukasawa Yoshitaka Ushiku 38 0 0 05 Aug 2024
Learning Video Context as Interleaved Multimodal Sequences S. Shao Pengchuan Zhang Y. Li Xide Xia A. Meso Ziteng Gao Jinheng Xie N. Holliman Mike Zheng Shou 41 5 0 31 Jul 2024
OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text Qingyun Li Zhe Chen Weiyun Wang Wenhai Wang Shenglong Ye ... Dahua Lin Yu Qiao Botian Shi Conghui He Jifeng Dai VLM OffRL 48 19 0 12 Jun 2024
Encoding and Controlling Global Semantics for Long-form Video Question Answering Thong Nguyen Zhiyuan Hu Xiaobao Wu Cong-Duy Nguyen See-Kiong Ng A. Luu 35 2 0 30 May 2024
CLIP-Powered TASS: Target-Aware Single-Stream Network for Audio-Visual Question Answering Yuanyuan Jiang Jianqin Yin 38 1 0 13 May 2024
Unified Video-Language Pre-training with Synchronized Audio Shentong Mo Haofan Wang Huaxia Li Xu Tang 30 2 0 12 May 2024
Sequential Compositional Generalization in Multimodal Models Semih Yagcioglu Osman Batur .Ince Aykut Erdem Erkut Erdem Desmond Elliott Deniz Yuret 34 1 0 18 Apr 2024
Streaming Dense Video Captioning Xingyi Zhou Anurag Arnab Shyamal Buch Shen Yan Austin Myers Xuehan Xiong Arsha Nagrani Cordelia Schmid VLM 29 30 0 01 Apr 2024
LocCa: Visual Pretraining with Location-aware Captioners Bo Wan Michael Tschannen Yongqin Xian Filip Pavetić Ibrahim M. Alabdulmohsin Xiao Wang André Susano Pinto Andreas Steiner Lucas Beyer Xiao-Qi Zhai VLM 40 5 0 28 Mar 2024
Towards Multimodal Video Paragraph Captioning Models Robust to Missing Modality Sishuo Chen Lei Li Shuhuai Ren Rundong Gao Yuanxin Liu Xiaohan Bi Xu Sun Lu Hou 27 3 0 28 Mar 2024
OmniVid: A Generative Framework for Universal Video Understanding Junke Wang Dongdong Chen Chong Luo Bo He Lu Yuan Zuxuan Wu Yu-Gang Jiang VLM VGen 69 14 0 26 Mar 2024
InternVideo2: Scaling Video Foundation Models for Multimodal Video Understanding Yi Wang Kunchang Li Xinhao Li Jiashuo Yu Yinan He ... Hongjie Zhang Yifei Huang Yu Qiao Yali Wang Limin Wang 27 44 0 22 Mar 2024
VidLA: Video-Language Alignment at Scale Mamshad Nayeem Rizve Fan Fei Jayakrishnan Unnikrishnan Son Tran Benjamin Z. Yao Belinda Zeng Mubarak Shah Trishul M. Chilimbi VLM AI4TS 43 4 0 21 Mar 2024
Annotations on a Budget: Leveraging Geo-Data Similarity to Balance Model Performance and Annotation Cost Oana Ignat Longju Bai Joan Nwatu Rada Mihalcea 36 6 0 12 Mar 2024
Answering Diverse Questions via Text Attached with Key Audio-Visual Clues Qilang Ye Zitong Yu Xin Liu 33 1 0 11 Mar 2024
VideoPrism: A Foundational Visual Encoder for Video Understanding Long Zhao N. B. Gundavarapu Liangzhe Yuan Hao Zhou Shen Yan ... Huisheng Wang Hartwig Adam Mikhail Sirotenko Ting Liu Boqing Gong VGen 27 29 0 20 Feb 2024
Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoning Long Qian Juncheng Billy Li Yu-hao Wu Yaobo Ye Hao Fei Tat-Seng Chua Yueting Zhuang Siliang Tang MLLM LRM 60 47 0 18 Feb 2024
Revisiting Feature Prediction for Learning Visual Representations from Video Adrien Bardes Q. Garrido Jean Ponce Xinlei Chen Michael G. Rabbat Yann LeCun Mahmoud Assran Nicolas Ballas MDE VLM 82 70 0 15 Feb 2024
CREMA: Generalizable and Efficient Video-Language Reasoning via Multimodal Modular Fusion Shoubin Yu Jaehong Yoon Mohit Bansal 77 4 0 08 Feb 2024
Can MLLMs Perform Text-to-Image In-Context Learning? Yuchen Zeng Wonjun Kang Yicong Chen Hyung Il Koo Kangwook Lee MLLM 23 9 0 02 Feb 2024
SNP-S3: Shared Network Pre-training and Significant Semantic Strengthening for Various Video-Text Tasks Xingning Dong Qingpei Guo Tian Gan Qing Wang Jianlong Wu Xiangyuan Ren Yuan-Chia Cheng Wei Chu 21 5 0 31 Jan 2024
MM-LLMs: Recent Advances in MultiModal Large Language Models Duzhen Zhang Yahan Yu Jiahua Dong Chenxing Li Dan Su Chenhui Chu Dong Yu OffRL LRM 37 173 0 24 Jan 2024
Seeing the Unseen: Visual Common Sense for Semantic Placement Ram Ramrakhya Aniruddha Kembhavi Dhruv Batra Z. Kira Kuo-Hao Zeng Luca Weihs VLM 33 4 0 15 Jan 2024
Distilling Vision-Language Models on Millions of Videos Yue Zhao Long Zhao Xingyi Zhou Jialin Wu Chun-Te Chu ... Hartwig Adam Ting Liu Boqing Gong Philipp Krahenbuhl Liangzhe Yuan VLM 21 13 0 11 Jan 2024
Efficient Multiscale Multimodal Bottleneck Transformer for Audio-Video Classification Wentao Zhu 22 5 0 08 Jan 2024
Efficient Selective Audio Masked Multimodal Bottleneck Transformer for Audio-Video Classification Wentao Zhu 19 4 0 08 Jan 2024
STAIR: Spatial-Temporal Reasoning with Auditable Intermediate Results for Video Question Answering Yueqian Wang Yuxuan Wang Kai Chen Dongyan Zhao 25 2 0 08 Jan 2024
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action Jiasen Lu Christopher Clark Sangho Lee Zichen Zhang Savya Khosla Ryan Marten Derek Hoiem Aniruddha Kembhavi VLM MLLM 27 143 0 28 Dec 2023
ViLA: Efficient Video-Language Alignment for Video Question Answering Xijun Wang Junbang Liang Chun-Kai Wang Kenan Deng Yu Lou Ming-Chyuan Lin Shan Yang 24 13 0 13 Dec 2023
Localized Symbolic Knowledge Distillation for Visual Commonsense Models J. Park Jack Hessel Khyathi Raghavi Chandu Paul Pu Liang Ximing Lu ... Youngjae Yu Qiuyuan Huang Jianfeng Gao Ali Farhadi Yejin Choi VLM 16 11 0 08 Dec 2023