MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound

7 January 2022

Yejin Choi

Papers citing "MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound"

42 / 42 papers shown

Title
TEMPURA: Temporal Event Masked Prediction and Understanding for Reasoning in Action Jen-Hao Cheng Vivian Wang Huayu Wang Huapeng Zhou Yi-Hao Peng ... Wenhao Chai Yi-Ling Chen Vibhav Vineet Qin Cai Jenq-Neng Hwang AI4TS 35 0 0 02 May 2025
Learning Streaming Video Representation via Multitask Training Yibin Yan Jilan Xu Shangzhe Di Yikun Liu Yudi Shi Qirui Chen Zeqian Li Yifei Huang Weidi Xie CLL 76 0 0 28 Apr 2025
2.5 Years in Class: A Multimodal Textbook for Vision-Language Pretraining Wenqi Zhang Hang Zhang Xin Li Jiashuo Sun Yongliang Shen Weiming Lu Deli Zhao Yueting Zhuang Lidong Bing VLM 37 2 0 01 Jan 2025
Neptune: The Long Orbit to Benchmarking Long Video Understanding Arsha Nagrani Mingda Zhang Ramin Mehran Rachel Hornung N. B. Gundavarapu ... Boqing Gong Cordelia Schmid Mikhail Sirotenko Yukun Zhu Tobias Weyand 98 4 0 12 Dec 2024
Video LLMs for Temporal Reasoning in Long Videos Fawad Javed Fateh Umer Ahmed Hamza Khan M. Zia Quoc-Huy Tran VLM 77 0 0 04 Dec 2024
The Sound of Water: Inferring Physical Properties from Pouring Liquids Piyush Bagad Makarand Tapaswi Cees G. M. Snoek Andrew Zisserman 37 0 0 18 Nov 2024
TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuning Xiangyu Zeng Kunchang Li Chenting Wang Xinhao Li Tianxiang Jiang ... Zhengrong Yue Yi Wang Yali Wang Yu Qiao Limin Wang MLLM VLM AI4TS 57 14 0 25 Oct 2024
Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understanding Yunze Man Shuhong Zheng Zhipeng Bao M. Hebert Liang-Yan Gui Yu-xiong Wang 70 15 0 05 Sep 2024
Learning Video Context as Interleaved Multimodal Sequences S. Shao Pengchuan Zhang Y. Li Xide Xia A. Meso Ziteng Gao Jinheng Xie N. Holliman Mike Zheng Shou 41 5 0 31 Jul 2024
Encoding and Controlling Global Semantics for Long-form Video Question Answering Thong Nguyen Zhiyuan Hu Xiaobao Wu Cong-Duy Nguyen See-Kiong Ng A. Luu 32 2 0 30 May 2024
CLIP-Powered TASS: Target-Aware Single-Stream Network for Audio-Visual Question Answering Yuanyuan Jiang Jianqin Yin 30 1 0 13 May 2024
CREMA: Generalizable and Efficient Video-Language Reasoning via Multimodal Modular Fusion Shoubin Yu Jaehong Yoon Mohit Bansal 67 4 0 08 Feb 2024
Quilt-LLaVA: Visual Instruction Tuning by Extracting Localized Narratives from Open-Source Histopathology Videos M. S. Seyfioglu Wisdom O. Ikezogwo Fatemeh Ghezloo Ranjay Krishna Linda G. Shapiro 22 31 0 07 Dec 2023
Spacewalk-18: A Benchmark for Multimodal and Long-form Procedural Video Understanding in Novel Domains Rohan Myer Krishnan Zitian Tang Zhiqiu Yu Chen Sun 35 1 0 30 Nov 2023
LEAP: LLM-Generation of Egocentric Action Programs Eadom Dessalene Michael Maynord Cornelia Fermuller Yiannis Aloimonos 16 3 0 29 Nov 2023
Zero-Shot and Few-Shot Video Question Answering with Multi-Modal Prompts Bipin Rajendran Bashir M. Al-Hashimi MLLM VLM 26 2 0 27 Sep 2023
Quilt-1M: One Million Image-Text Pairs for Histopathology Wisdom O. Ikezogwo M. S. Seyfioglu Fatemeh Ghezloo Dylan Stefan Chan Geva Fatwir Sheikh Mohammed Pavan Kumar Anand Ranjay Krishna Linda G. Shapiro CLIP VLM 125 101 0 20 Jun 2023
IdealGPT: Iteratively Decomposing Vision and Language Reasoning via Large Language Models Haoxuan You Rui Sun Zhecan Wang Long Chen Gengyu Wang Hammad A. Ayyubi Kai-Wei Chang Shih-Fu Chang VLM MLLM LRM 35 42 0 24 May 2023
Difference-Masking: Choosing What to Mask in Continued Pretraining Alex Wilf Syeda Nahida Akter Leena Mathur Paul Pu Liang Sheryl Mathew Mengrou Shou Eric Nyberg Louis-Philippe Morency CLL SSL 19 4 0 23 May 2023
Self-Chained Image-Language Model for Video Localization and Question Answering Shoubin Yu Jaemin Cho Prateek Yadav Mohit Bansal 31 129 0 11 May 2023
Procedure-Aware Pretraining for Instructional Video Understanding Honglu Zhou Roberto Martín-Martín Mubbasir Kapadia Silvio Savarese Juan Carlos Niebles 20 38 0 31 Mar 2023
See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning Zhenfang Chen Qinhong Zhou Yikang Shen Yining Hong Hao Zhang Chuang Gan LRM VLM 29 35 0 12 Jan 2023
Leveraging the Video-level Semantic Consistency of Event for Audio-visual Event Localization Yuanyuan Jiang Jianqin Yin Yonghao Dang 27 4 0 11 Oct 2022
VIMA: General Robot Manipulation with Multimodal Prompts Yunfan Jiang Agrim Gupta Zichen Zhang Guanzhi Wang Yongqiang Dou Yanjun Chen Li Fei-Fei Anima Anandkumar Yuke Zhu Linxi Fan LM&Ro 15 332 0 06 Oct 2022
TVLT: Textless Vision-Language Transformer Zineng Tang Jaemin Cho Yixin Nie Mohit Bansal VLM 38 28 0 28 Sep 2022
UAVM: Towards Unifying Audio and Visual Models Yuan Gong Alexander H. Liu Andrew Rouditchenko James R. Glass 19 20 0 29 Jul 2022
Zero-Shot Video Question Answering via Frozen Bidirectional Language Models Antoine Yang Antoine Miech Josef Sivic Ivan Laptev Cordelia Schmid 17 225 0 16 Jun 2022
i-Code: An Integrative and Composable Multimodal Learning Framework Ziyi Yang Yuwei Fang Chenguang Zhu Reid Pryzant Dongdong Chen ... Bin Xiao Yuanxun Lu Takuya Yoshioka Michael Zeng Xuedong Huang 35 45 0 03 May 2022
ECLIPSE: Efficient Long-range Video Retrieval using Sight and Sound Yan-Bo Lin Jie Lei Mohit Bansal Gedas Bertasius 23 39 0 06 Apr 2022
VLP: A Survey on Vision-Language Pre-training Feilong Chen Duzhen Zhang Minglun Han Xiuyi Chen Jing Shi Shuang Xu Bo Xu VLM 79 208 0 18 Feb 2022
VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding Hu Xu Gargi Ghosh Po-Yao (Bernie) Huang Dmytro Okhonko Armen Aghajanyan Florian Metze Luke Zettlemoyer Florian Metze Luke Zettlemoyer Christoph Feichtenhofer CLIP VLM 245 554 0 28 Sep 2021
Multimodal Self-Supervised Learning of General Audio Representations Luyu Wang Pauline Luc Adrià Recasens Jean-Baptiste Alayrac Aaron van den Oord SSL 70 41 0 26 Apr 2021
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text Hassan Akbari Liangzhe Yuan Rui Qian Wei-Hong Chuang Shih-Fu Chang Yin Cui Boqing Gong ViT 231 573 0 22 Apr 2021
Carbon Emissions and Large Neural Network Training David A. Patterson Joseph E. Gonzalez Quoc V. Le Chen Liang Lluís-Miquel Munguía D. Rothchild David R. So Maud Texier J. Dean AI4CE 239 626 0 21 Apr 2021
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision Chao Jia Yinfei Yang Ye Xia Yi-Ting Chen Zarana Parekh Hieu H. Pham Quoc V. Le Yun-hsuan Sung Zhen Li Tom Duerig VLM CLIP 293 3,683 0 11 Feb 2021
Is Space-Time Attention All You Need for Video Understanding? Gedas Bertasius Heng Wang Lorenzo Torresani ViT 278 1,939 0 09 Feb 2021
Unifying Vision-and-Language Tasks via Text Generation Jaemin Cho Jie Lei Hao Tan Mohit Bansal MLLM 249 518 0 04 Feb 2021
Disembodied Machine Learning: On the Illusion of Objectivity in NLP Zeerak Talat Smarika Lulz Joachim Bingel Isabelle Augenstein 88 51 0 28 Jan 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 245 1,977 0 31 Dec 2020
Extracting Training Data from Large Language Models Nicholas Carlini Florian Tramèr Eric Wallace Matthew Jagielski Ariel Herbert-Voss ... Tom B. Brown D. Song Ulfar Erlingsson Alina Oprea Colin Raffel MLAU SILM 267 1,798 0 14 Dec 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 220 4,424 0 23 Jan 2020
Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models Cheolhyoung Lee Kyunghyun Cho Wanmo Kang MoE 225 204 0 25 Sep 2019