MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound

7 January 2022

Yejin Choi

Papers citing "MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound"

50 / 163 papers shown

Title
Quilt-LLaVA: Visual Instruction Tuning by Extracting Localized Narratives from Open-Source Histopathology Videos M. S. Seyfioglu Wisdom O. Ikezogwo Fatemeh Ghezloo Ranjay Krishna Linda G. Shapiro 24 33 0 07 Dec 2023
TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding Shuhuai Ren Linli Yao Shicheng Li Xu Sun Lu Hou VLM MLLM 10 174 0 04 Dec 2023
Zero-Shot Video Question Answering with Procedural Programs Rohan Choudhury Koichiro Niinuma Kris M. Kitani László A. Jeni 19 21 0 01 Dec 2023
CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generation Zineng Tang Ziyi Yang Mahmoud Khademi Yang Liu Chenguang Zhu Mohit Bansal LRM MLLM AuLLM 52 44 0 30 Nov 2023
Spacewalk-18: A Benchmark for Multimodal and Long-form Procedural Video Understanding in Novel Domains Rohan Myer Krishnan Zitian Tang Zhiqiu Yu Chen Sun 40 1 0 30 Nov 2023
LEAP: LLM-Generation of Egocentric Action Programs Eadom Dessalene Michael Maynord Cornelia Fermuller Yiannis Aloimonos 16 3 0 29 Nov 2023
Efficient Pre-training for Localized Instruction Generation of Videos Anil Batra Davide Moltisanti Laura Sevilla-Lara Marcus Rohrbach Frank Keller 17 0 0 27 Nov 2023
Vamos: Versatile Action Models for Video Understanding Shijie Wang Qi Zhao Minh Quan Do Nakul Agarwal Kwonjoon Lee Chen Sun 27 19 0 22 Nov 2023
Vision-Language Integration in Multimodal Video Transformers (Partially) Aligns with the Brain Dota Tianai Dong Mariya Toneva 14 3 0 13 Nov 2023
Mirasol3B: A Multimodal Autoregressive model for time-aligned and contextual modalities A. Piergiovanni Isaac Noble Dahun Kim Michael S. Ryoo Victor Gomes A. Angelova 33 19 0 09 Nov 2023
ACQUIRED: A Dataset for Answering Counterfactual Questions In Real-Life Videos Te-Lin Wu Zi-Yi Dou Qingyuan Hu Yu Hou Nischal Reddy Chandra Marjorie Freedman R. Weischedel Nanyun Peng 24 5 0 02 Nov 2023
Long Story Short: a Summarize-then-Search Method for Long Video Question Answering Jiwan Chung Youngjae Yu 95 5 0 02 Nov 2023
Can Foundation Models Watch, Talk and Guide You Step by Step to Make a Cake? Yuwei Bao Keunwoo Peter Yu Yichi Zhang Shane Storks Itamar Bar-Yossef Alexander De La Iglesia Megan Su Xiao Lin Zheng Joyce Chai 35 8 0 01 Nov 2023
Defining a New NLP Playground Sha Li Chi Han Pengfei Yu Carl N. Edwards Manling Li ... Yi Ren Fung Charles Yu Joel R. Tetreault Eduard H. Hovy Heng Ji 31 5 0 31 Oct 2023
Disentangled Counterfactual Learning for Physical Audiovisual Commonsense Reasoning Changsheng Lv Shuai Zhang Yapeng Tian Mengshi Qi Huadong Ma CML 37 16 0 30 Oct 2023
Harvest Video Foundation Models via Efficient Post-Pretraining Yizhuo Li Kunchang Li Yinan He Yi Wang Yali Wang Limin Wang Yu Qiao Ping Luo CLIP VLM VGen 33 2 0 30 Oct 2023
Query-aware Long Video Localization and Relation Discrimination for Deep Video Understanding Yuanxing Xu Yuting Wei Bin Wu 22 0 0 19 Oct 2023
Separating Invisible Sounds Toward Universal Audiovisual Scene-Aware Sound Separation Yiyang Su A. Vosoughi Shijian Deng Yapeng Tian Chenliang Xu 26 4 0 18 Oct 2023
VidCoM: Fast Video Comprehension through Large Language Models with Multimodal Tools Huihui Gong Minjing Dong Siqi Ma S. Çamtepe Chang Xu Lei Hou Surya Nepal VLM MLLM 45 0 0 16 Oct 2023
Analyzing Zero-Shot Abilities of Vision-Language Models on Video Understanding Tasks Avinash Madasu Anahita Bhiwandiwalla Vasudev Lal VLM 29 0 0 07 Oct 2023
SelfGraphVQA: A Self-Supervised Graph Neural Network for Scene-based Question Answering Bruno Souza Marius Aasan Hélio Pedrini Adín Ramirez Rivera SSL 14 2 0 03 Oct 2023
Zero-Shot and Few-Shot Video Question Answering with Multi-Modal Prompts Bipin Rajendran Bashir M. Al-Hashimi MLLM VLM 26 2 0 27 Sep 2023
MUTEX: Learning Unified Policies from Multimodal Task Specifications Rutav Shah Roberto Martín-Martín Yuke Zhu OffRL 39 54 0 25 Sep 2023
VidChapters-7M: Video Chapters at Scale Antoine Yang Arsha Nagrani Ivan Laptev Josef Sivic Cordelia Schmid VGen 13 26 0 25 Sep 2023
Unified Coarse-to-Fine Alignment for Video-Text Retrieval Ziyang Wang Yi-Lin Sung Feng Cheng Gedas Bertasius Mohit Bansal 93 44 0 18 Sep 2023
Can I Trust Your Answer? Visually Grounded Video Question Answering Junbin Xiao Angela Yao Yicong Li Tat-Seng Chua 28 46 0 04 Sep 2023
CoVR: Learning Composed Video Retrieval from Web Video Captions Lucas Ventura Antoine Yang Cordelia Schmid Gül Varol 17 27 0 28 Aug 2023
Simple Baselines for Interactive Video Retrieval with Questions and Answers Kaiqu Liang Samuel Albanie 22 2 0 21 Aug 2023
Long-range Multimodal Pretraining for Movie Understanding Dawit Mureja Argaw Joon-Young Lee Markus Woodson In So Kweon Fabian Caba Heilbron VLM 25 7 0 18 Aug 2023
Bridging High-Quality Audio and Video via Language for Sound Effects Retrieval from Visual Queries J. Wilkins Justin Salamon Magdalena Fuentes J. P. Bello Oriol Nieto CLIP 14 5 0 17 Aug 2023
Tem-adapter: Adapting Image-Text Pretraining for Video Question Answer Guangyi Chen Xiao Liu Guangrun Wang Kun Zhang Philip H.S.Torr Xiaoping Zhang Yansong Tang 19 17 0 16 Aug 2023
OpenProteinSet: Training data for structural biology at scale Gustaf Ahdritz N. Bouatta S. Kadyan Lukas Jarosch Daniel Berenberg Ian Fisk Andrew Watkins Stephen Ra Richard Bonneau Mohammed AlQuraishi AI4CE 6 10 0 10 Aug 2023
UnIVAL: Unified Model for Image, Video, Audio and Language Tasks Mustafa Shukor Corentin Dancette Alexandre Ramé Matthieu Cord MoMe MLLM 27 42 0 30 Jul 2023
Audio-Enhanced Text-to-Video Retrieval using Text-Conditioned Feature Alignment Sarah Ibrahimi Xiaohang Sun Pichao Wang Amanmeet Garg Ashutosh Sanan Mohamed Omar 44 14 0 24 Jul 2023
InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation Yi Wang Yinan He Yizhuo Li Kunchang Li Jiashuo Yu ... Ping Luo Ziwei Liu Yali Wang Limin Wang Yu Qiao VLM VGen 25 244 0 13 Jul 2023
Emu: Generative Pretraining in Multimodality Quan-Sen Sun Qiying Yu Yufeng Cui Fan Zhang Xiaosong Zhang Yueze Wang Hongcheng Gao Jingjing Liu Tiejun Huang Xinlong Wang MLLM 27 126 0 11 Jul 2023
Read, Look or Listen? What's Needed for Solving a Multimodal Dataset Netta Madvil Yonatan Bitton Roy Schwartz 16 2 0 06 Jul 2023
Exploring the Role of Audio in Video Captioning Yuhan Shen Linjie Yang Longyin Wen Haichao Yu Ehsan Elhamifar Heng Wang 18 2 0 21 Jun 2023
Quilt-1M: One Million Image-Text Pairs for Histopathology Wisdom O. Ikezogwo M. S. Seyfioglu Fatemeh Ghezloo Dylan Stefan Chan Geva Fatwir Sheikh Mohammed Pavan Kumar Anand Ranjay Krishna Linda G. Shapiro CLIP VLM 125 109 0 20 Jun 2023
Meta-Personalizing Vision-Language Models to Find Named Instances in Video Chun-Hsiao Yeh Bryan C. Russell Josef Sivic Fabian Caba Heilbron Simon Jenni VLM MLLM 44 9 0 16 Jun 2023
Retrieving-to-Answer: Zero-Shot Video Question Answering with Frozen Large Language Models Junting Pan Ziyi Lin Yuying Ge Xiatian Zhu Renrui Zhang Yi Wang Yu Qiao Hongsheng Li MLLM 24 26 0 15 Jun 2023
Language-Guided Music Recommendation for Video via Prompt Analogies Daniel McKee Justin Salamon Josef Sivic Bryan C. Russell VGen 23 26 0 15 Jun 2023
DocumentCLIP: Linking Figures and Main Body Text in Reflowed Documents Fuxiao Liu Hao Tan Chris Tensmeyer CLIP VLM 20 18 0 09 Jun 2023
Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applications Paul Pu Liang Chun Kai Ling Yun Cheng A. Obolenskiy Yudong Liu Rohan Pandey Alex Wilf Louis-Philippe Morency Ruslan Salakhutdinov OffRL 28 11 0 07 Jun 2023
VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset Sihan Chen Handong Li Qunbo Wang Zijia Zhao Ming-Ting Sun Xinxin Zhu J. Liu 30 95 0 29 May 2023
IdealGPT: Iteratively Decomposing Vision and Language Reasoning via Large Language Models Haoxuan You Rui Sun Zhecan Wang Long Chen Gengyu Wang Hammad A. Ayyubi Kai-Wei Chang Shih-Fu Chang VLM MLLM LRM 39 43 0 24 May 2023
Difference-Masking: Choosing What to Mask in Continued Pretraining Alex Wilf Syeda Nahida Akter Leena Mathur Paul Pu Liang Sheryl Mathew Mengrou Shou Eric Nyberg Louis-Philippe Morency CLL SSL 19 4 0 23 May 2023
Has It All Been Solved? Open NLP Research Questions Not Solved by Large Language Models Oana Ignat Zhijing Jin Artem Abzaliev Laura Biester Santiago Castro ... Verónica Pérez-Rosas Siqi Shen Zekun Wang Winston Wu Rada Mihalcea LRM 24 6 0 21 May 2023
Target-Aware Spatio-Temporal Reasoning via Answering Questions in Dynamics Audio-Visual Scenarios Yuanyuan Jiang Jianqin Yin 10 6 0 21 May 2023
i-Code V2: An Autoregressive Generation Framework over Vision, Language, and Speech Data Ziyi Yang Mahmoud Khademi Yichong Xu Reid Pryzant Yuwei Fang ... Yu Shi Lu Yuan Takuya Yoshioka Michael Zeng Xuedong Huang 17 2 0 21 May 2023