MERLOT: Multimodal Neural Script Knowledge Models

4 June 2021

Yejin Choi

Papers citing "MERLOT: Multimodal Neural Script Knowledge Models"

48 / 48 papers shown

Title
VAEmo: Efficient Representation Learning for Visual-Audio Emotion with Knowledge Injection Hao Cheng Zhiwei Zhao Yichao He Zhenzhen Hu Jia Li M. Wang Richang Hong 36 0 0 05 May 2025
Raccoon: Multi-stage Diffusion Training with Coarse-to-Fine Curating Videos Zhiyu Tan Junyan Wang Hao Yang Luozheng Qin Hesen Chen Qiang-feng Zhou Hao Li VGen 64 0 0 28 Feb 2025
Pretrained Image-Text Models are Secretly Video Captioners Chunhui Zhang Yiren Jian Z. Ouyang Soroush Vosoughi VLM 63 3 0 20 Feb 2025
A Simple Recipe for Contrastively Pre-training Video-First Encoders Beyond 16 Frames Pinelopi Papalampidi Skanda Koppula Shreya Pathak Justin T Chiu Joseph Heyward Viorica Patraucean Jiajun Shen Antoine Miech Andrew Zisserman Aida Nematzdeh VLM 56 23 0 31 Dec 2024
Neptune: The Long Orbit to Benchmarking Long Video Understanding Arsha Nagrani Mingda Zhang Ramin Mehran Rachel Hornung N. B. Gundavarapu ... Boqing Gong Cordelia Schmid Mikhail Sirotenko Yukun Zhu Tobias Weyand 98 4 0 12 Dec 2024
Can LVLMs Describe Videos like Humans? A Five-in-One Video Annotations Benchmark for Better Human-Machine Comparison Shiyu Hu Xuchen Li X. Li Jing Zhang Yipei Wang Xin Zhao Kang Hao Cheong VLM 22 1 0 20 Oct 2024
Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Content Qiuheng Wang Yukai Shi Jiarong Ou R. J. Chen Ke Lin ... Mingwu Zheng Xin Tao Fei Yang Pengfei Wan Di Zhang VGen 83 18 0 10 Oct 2024
Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentation Kun Yuan V. Srivastav Nassir Navab N. Padoy 41 7 0 30 Sep 2024
MIO: A Foundation Model on Multimodal Tokens Zekun Wang King Zhu Chunpu Xu Wangchunshu Zhou Jiaheng Liu ... Yuanxing Zhang Ge Zhang Ke Xu Jie Fu Wenhao Huang MLLM AuLLM 44 11 0 26 Sep 2024
Ego-VPA: Egocentric Video Understanding with Parameter-efficient Adaptation Tz-Ying Wu Kyle Min Subarna Tripathi Nuno Vasconcelos EgoV 51 0 0 28 Jul 2024
Detecting Multimodal Situations with Insufficient Context and Abstaining from Baseless Predictions Junzhang Liu Zhecan Wang Hammad A. Ayyubi Haoxuan You Chris Thomas Rui Sun Shih-Fu Chang Kai-Wei Chang 21 0 0 18 May 2024
VideoDistill: Language-aware Vision Distillation for Video Question Answering Bo Zou Chao Yang Yu Qiao Chengbin Quan Youjian Zhao VGen 33 1 0 01 Apr 2024
EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World Yifei Huang Guo Chen Jilan Xu Mingfang Zhang Lijin Yang ... Hongjie Zhang Lu Dong Yali Wang Limin Wang Yu Qiao EgoV 49 32 0 24 Mar 2024
Sports-QA: A Large-Scale Video Question Answering Benchmark for Complex and Professional Sports Haopeng Li Andong Deng Qiuhong Ke Jun Liu Hossein Rahmani Yulan Guo Mohammed Bennamoun Chen Chen 37 17 0 03 Jan 2024
Quilt-LLaVA: Visual Instruction Tuning by Extracting Localized Narratives from Open-Source Histopathology Videos M. S. Seyfioglu Wisdom O. Ikezogwo Fatemeh Ghezloo Ranjay Krishna Linda G. Shapiro 22 31 0 07 Dec 2023
Spacewalk-18: A Benchmark for Multimodal and Long-form Procedural Video Understanding in Novel Domains Rohan Myer Krishnan Zitian Tang Zhiqiu Yu Chen Sun 35 1 0 30 Nov 2023
The curse of language biases in remote sensing VQA: the role of spatial attributes, language diversity, and the need for clear evaluation Christel Chappuis Eliot Walt Vincent Mendez Sylvain Lobry B. L. Saux D. Tuia 15 3 0 28 Nov 2023
Zero-Shot and Few-Shot Video Question Answering with Multi-Modal Prompts Bipin Rajendran Bashir M. Al-Hashimi MLLM VLM 26 2 0 27 Sep 2023
GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest Shilong Zhang Pei Sun Shoufa Chen Min Xiao Wenqi Shao Wenwei Zhang Yu Liu Kai-xiang Chen Ping Luo VLM MLLM 80 222 0 07 Jul 2023
Quilt-1M: One Million Image-Text Pairs for Histopathology Wisdom O. Ikezogwo M. S. Seyfioglu Fatemeh Ghezloo Dylan Stefan Chan Geva Fatwir Sheikh Mohammed Pavan Kumar Anand Ranjay Krishna Linda G. Shapiro CLIP VLM 125 101 0 20 Jun 2023
Procedure-Aware Pretraining for Instructional Video Understanding Honglu Zhou Roberto Martín-Martín Mubbasir Kapadia Silvio Savarese Juan Carlos Niebles 20 38 0 31 Mar 2023
MaMMUT: A Simple Architecture for Joint Learning for MultiModal Tasks Weicheng Kuo A. Piergiovanni Dahun Kim Xiyang Luo Benjamin Caine ... Luowei Zhou Andrew M. Dai Zhifeng Chen Claire Cui A. Angelova MLLM VLM 12 23 0 29 Mar 2023
Video-Text as Game Players: Hierarchical Banzhaf Interaction for Cross-Modal Representation Learning Peng Jin Jinfa Huang Pengfei Xiong Shangxuan Tian Chang-rui Liu Xiang Ji Li-ming Yuan Jie Chen 23 48 0 25 Mar 2023
CleanCLIP: Mitigating Data Poisoning Attacks in Multimodal Contrastive Learning Hritik Bansal Nishad Singhi Yu Yang Fan Yin Aditya Grover Kai-Wei Chang AAML 24 41 0 06 Mar 2023
STOA-VLP: Spatial-Temporal Modeling of Object and Action for Video-Language Pre-training Weihong Zhong Mao Zheng Duyu Tang Xuan Luo Heng Gong Xiaocheng Feng Bing Qin 22 8 0 20 Feb 2023
Towards a Unified Model for Generating Answers and Explanations in Visual Question Answering Chenxi Whitehouse Tillman Weyde Pranava Madhyastha LRM 30 3 0 25 Jan 2023
Curriculum Script Distillation for Multilingual Visual Question Answering Khyathi Raghavi Chandu A. Geramifard 17 0 0 17 Jan 2023
Perceive, Ground, Reason, and Act: A Benchmark for General-purpose Visual Representation Jiangyong Huang William Zhu Baoxiong Jia Zan Wang Xiaojian Ma Qing Li Siyuan Huang 24 5 0 28 Nov 2022
SMAUG: Sparse Masked Autoencoder for Efficient Video-Language Pre-training Yuanze Lin Chen Wei Huiyu Wang Alan Yuille Cihang Xie 3DGS 26 15 0 21 Nov 2022
VIMA: General Robot Manipulation with Multimodal Prompts Yunfan Jiang Agrim Gupta Zichen Zhang Guanzhi Wang Yongqiang Dou Yanjun Chen Li Fei-Fei Anima Anandkumar Yuke Zhu Linxi Fan LM&Ro 15 332 0 06 Oct 2022
TVLT: Textless Vision-Language Transformer Zineng Tang Jaemin Cho Yixin Nie Mohit Bansal VLM 38 28 0 28 Sep 2022
Video Question Answering with Iterative Video-Text Co-Tokenization A. Piergiovanni K. Morton Weicheng Kuo Michael S. Ryoo A. Angelova 13 17 0 01 Aug 2022
Clover: Towards A Unified Video-Language Alignment and Fusion Model Jingjia Huang Yinan Li Jiashi Feng Xinglong Wu Xiaoshuai Sun Rongrong Ji VLM 19 46 0 16 Jul 2022
Zero-Shot Video Question Answering via Frozen Bidirectional Language Models Antoine Yang Antoine Miech Josef Sivic Ivan Laptev Cordelia Schmid 17 225 0 16 Jun 2022
Multimodal Learning with Transformers: A Survey P. Xu Xiatian Zhu David A. Clifton ViT 41 518 0 13 Jun 2022
CyCLIP: Cyclic Contrastive Language-Image Pretraining Shashank Goel Hritik Bansal S. Bhatia Ryan A. Rossi Vishwa Vinay Aditya Grover CLIP VLM 166 131 0 28 May 2022
Learning to Answer Visual Questions from Web Videos Antoine Yang Antoine Miech Josef Sivic Ivan Laptev Cordelia Schmid ViT 22 33 0 10 May 2022
i-Code: An Integrative and Composable Multimodal Learning Framework Ziyi Yang Yuwei Fang Chenguang Zhu Reid Pryzant Dongdong Chen ... Bin Xiao Yuanxun Lu Takuya Yoshioka Michael Zeng Xuedong Huang 35 45 0 03 May 2022
Are Multimodal Transformers Robust to Missing Modality? Mengmeng Ma Jian Ren Long Zhao Davide Testuggine Xi Peng ViT 26 145 0 12 Apr 2022
CLIP-TD: CLIP Targeted Distillation for Vision-Language Tasks Zhecan Wang Noel Codella Yen-Chun Chen Luowei Zhou Jianwei Yang Xiyang Dai Bin Xiao Haoxuan You Shih-Fu Chang Lu Yuan CLIP VLM 16 39 0 15 Jan 2022
Object-aware Video-language Pre-training for Retrieval Alex Jinpeng Wang Yixiao Ge Guanyu Cai Rui Yan Xudong Lin Ying Shan Xiaohu Qie Mike Zheng Shou ViT VLM 15 79 0 01 Dec 2021
VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling Tsu-jui Fu Linjie Li Zhe Gan Kevin Qinghong Lin W. Wang Lijuan Wang Zicheng Liu VLM 34 215 0 24 Nov 2021
How Much Can CLIP Benefit Vision-and-Language Tasks? Sheng Shen Liunian Harold Li Hao Tan Mohit Bansal Anna Rohrbach Kai-Wei Chang Z. Yao Kurt Keutzer CLIP VLM MLLM 182 403 0 13 Jul 2021
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text Hassan Akbari Liangzhe Yuan Rui Qian Wei-Hong Chuang Shih-Fu Chang Yin Cui Boqing Gong ViT 231 573 0 22 Apr 2021
Disembodied Machine Learning: On the Illusion of Objectivity in NLP Zeerak Talat Smarika Lulz Joachim Bingel Isabelle Augenstein 88 51 0 28 Jan 2021
Learning Temporal Dynamics from Cycles in Narrated Video Dave Epstein Jiajun Wu Cordelia Schmid Chen Sun AI4TS 20 14 0 07 Jan 2021
Extracting Training Data from Large Language Models Nicholas Carlini Florian Tramèr Eric Wallace Matthew Jagielski Ariel Herbert-Voss ... Tom B. Brown D. Song Ulfar Erlingsson Alina Oprea Colin Raffel MLAU SILM 267 1,798 0 14 Dec 2020
Interaction Networks for Learning about Objects, Relations and Physics Peter W. Battaglia Razvan Pascanu Matthew Lai Danilo Jimenez Rezende Koray Kavukcuoglu AI4CE OCL PINN GNN 255 1,394 0 01 Dec 2016