Multimodal Few-Shot Learning with Frozen Language Models

25 June 2021

Papers citing "Multimodal Few-Shot Learning with Frozen Language Models"

50 / 532 papers shown

Title
Learning Customized Visual Models with Retrieval-Augmented Knowledge Haotian Liu Kilho Son Jianwei Yang Ce Liu Jianfeng Gao Yong Jae Lee Chunyuan Li VLM 38 53 0 17 Jan 2023
Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Models Zhiqiu Lin Samuel Yu Zhiyi Kuang Deepak Pathak Deva Ramana VLM 15 100 0 16 Jan 2023
See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning Zhenfang Chen Qinhong Zhou Yikang Shen Yining Hong Hao Zhang Chuang Gan LRM VLM 29 35 0 12 Jan 2023
A Survey on In-context Learning Qingxiu Dong Lei Li Damai Dai Ce Zheng Jingyuan Ma ... Zhiyong Wu Baobao Chang Xu Sun Lei Li Zhifang Sui ReLM AIMat 20 460 0 31 Dec 2022
From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Models Jiaxian Guo Junnan Li Dongxu Li A. M. H. Tiong Boyang Albert Li Dacheng Tao Steven C. H. Hoi VLM MLLM 24 107 0 21 Dec 2022
Tackling Ambiguity with Images: Improved Multimodal Machine Translation and Contrastive Evaluation Matthieu Futeral Cordelia Schmid Ivan Laptev Benoît Sagot Rachel Bawden 24 26 0 20 Dec 2022
Optimizing Prompts for Text-to-Image Generation Y. Hao Zewen Chi Li Dong Furu Wei 27 139 0 19 Dec 2022
Multimodal Prototype-Enhanced Network for Few-Shot Action Recognition Xin Ni Yong Liu Hao Wen Yatai Ji Jing Xiao Yujiu Yang 24 9 0 09 Dec 2022
Learning Video Representations from Large Language Models Yue Zhao Ishan Misra Philipp Krahenbuhl Rohit Girdhar VLM AI4TS 20 164 0 08 Dec 2022
General-Purpose In-Context Learning by Meta-Learning Transformers Louis Kirsch James Harrison Jascha Narain Sohl-Dickstein Luke Metz 29 72 0 08 Dec 2022
M-VADER: A Model for Diffusion with Multimodal Context Samuel Weinbach Marco Bellagente C. Eichenberg Andrew M. Dai R. Baldock Souradeep Nanda Bjorn Deiseroth Koen Oostermeijer H. Teufel Andres Felipe Cruz Salinas DiffM 27 11 0 06 Dec 2022
Adapted Multimodal BERT with Layer-wise Fusion for Sentiment Analysis Odysseas S. Chlapanis Georgios Paraskevopoulos Alexandros Potamianos 21 9 0 01 Dec 2022
Prototypical Fine-tuning: Towards Robust Performance Under Varying Data Sizes Yiqiao Jin Xiting Wang Y. Hao Yizhou Sun Xing Xie 28 11 0 24 Nov 2022
Knowledge Prompting for Few-shot Action Recognition Yuheng Shi Xinxiao Wu Hanxi Lin VLM 11 4 0 22 Nov 2022
CapEnrich: Enriching Caption Semantics for Web Images via Cross-modal Pre-trained Knowledge Linli Yao Wei-Neng Chen Qin Jin VLM 22 10 0 17 Nov 2022
PromptCap: Prompt-Guided Task-Aware Image Captioning Yushi Hu Hang Hua Zhengyuan Yang Weijia Shi Noah A. Smith Jiebo Luo 28 101 0 15 Nov 2022
Language models are good pathologists: using attention-based sequence reduction and text-pretrained transformers for efficient WSI classification Juan Pisula Katarzyna Bozek VLM MedIm 20 3 0 14 Nov 2022
Few-shot Multimodal Sentiment Analysis based on Multimodal Probabilistic Fusion Prompts Xiaocui Yang Shi Feng Daling Wang Pengfei Hong Soujanya Poria 21 19 0 12 Nov 2022
Continuous Prompt Tuning Based Textual Entailment Model for E-commerce Entity Typing Yibo Wang Congying Xia Guan Wang Philip Yu 11 6 0 04 Nov 2022
Understanding and Mitigating Overfitting in Prompt Tuning for Vision-Language Models Cheng Ma Yang Liu Jiankang Deng Lingxi Xie Weiming Dong Changsheng Xu VLM VPVLM 26 43 0 04 Nov 2022
Text-Only Training for Image Captioning using Noise-Injected CLIP David Nukrai Ron Mokady Amir Globerson VLM CLIP 49 69 0 01 Nov 2022
Open-vocabulary Semantic Segmentation with Frozen Vision-Language Models Chaofan Ma Yu-Hao Yang Yanfeng Wang Ya-Qin Zhang Weidi Xie VLM 21 48 0 27 Oct 2022
Towards Unifying Reference Expression Generation and Comprehension Duo Zheng Tao Kong Ya Jing Jiaan Wang Xiaojie Wang ObjD 27 6 0 24 Oct 2022
Visualizing the Obvious: A Concreteness-based Ensemble Model for Noun Property Prediction Yue Yang Artemis Panagopoulou Marianna Apidianaki Mark Yatskar Chris Callison-Burch 21 2 0 24 Oct 2022
Z-LaVI: Zero-Shot Language Solver Fueled by Visual Imagination Yue Yang Wenlin Yao Hongming Zhang Xiaoyang Wang Dong Yu Jianshu Chen VLM 39 21 0 21 Oct 2022
Prompting through Prototype: A Prototype-based Prompt Learning on Pretrained Vision-Language Models Yue Zhang Hongliang Fei Dingcheng Li Tan Yu Ping Li VPVLM VLM 15 9 0 19 Oct 2022
CPL: Counterfactual Prompt Learning for Vision and Language Models Xuehai He Diji Yang Weixi Feng Tsu-jui Fu Arjun Reddy Akula Varun Jampani P. Narayana Sugato Basu William Yang Wang X. Wang VPVLM VLM 43 15 0 19 Oct 2022
Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training A. M. H. Tiong Junnan Li Boyang Albert Li Silvio Savarese S. Hoi MLLM 27 101 0 17 Oct 2022
LAION-5B: An open large-scale dataset for training next generation image-text models Christoph Schuhmann Romain Beaumont Richard Vencu Cade Gordon Ross Wightman ... Srivatsa Kundurthy Katherine Crowson Ludwig Schmidt R. Kaczmarczyk J. Jitsev VLM MLLM CLIP 43 3,247 0 16 Oct 2022
MAPL: Parameter-Efficient Adaptation of Unimodal Pre-Trained Models for Vision-Language Few-Shot Prompting Oscar Manas Pau Rodríguez López Saba Ahmadi Aida Nematzadeh Yash Goyal Aishwarya Agrawal VLM VPVLM 13 48 0 13 Oct 2022
Text-Derived Knowledge Helps Vision: A Simple Cross-modal Distillation for Video-based Action Anticipation Sayontan Ghosh Tanvi Aggarwal Minh Hoai Niranjan Balasubramanian VLM 4 4 0 12 Oct 2022
Generating Executable Action Plans with Environmentally-Aware Language Models Maitrey Gramopadhye D. Szafir LM&Ro LLMAG 8 22 0 10 Oct 2022
Hierarchical3D Adapters for Long Video-to-text Summarization Pinelopi Papalampidi Mirella Lapata VGen 27 12 0 10 Oct 2022
VIMA: General Robot Manipulation with Multimodal Prompts Yunfan Jiang Agrim Gupta Zichen Zhang Guanzhi Wang Yongqiang Dou Yanjun Chen Li Fei-Fei Anima Anandkumar Yuke Zhu Linxi Fan LM&Ro 18 334 0 06 Oct 2022
PLOT: Prompt Learning with Optimal Transport for Vision-Language Models Guangyi Chen Weiran Yao Xiangchen Song Xinyue Li Yongming Rao Kun Zhang VPVLM VLM 6 62 0 03 Oct 2022
SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentation R. Ramos Bruno Martins Desmond Elliott Yova Kementchedjhieva VLM 28 86 0 30 Sep 2022
Linearly Mapping from Image to Text Space Jack Merullo Louis Castricato Carsten Eickhoff Ellie Pavlick VLM 159 104 0 30 Sep 2022
REST: REtrieve & Self-Train for generative action recognition Adrian Bulat Enrique Sanchez Brais Martínez Georgios Tzimiropoulos VLM 16 4 0 29 Sep 2022
Prompt-guided Scene Generation for 3D Zero-Shot Learning Majid Nasiri A. Cheraghian T. Chowdhury Sahar Ahmadi Morteza Saberi Shafin Rahman 3DPC DiffM 11 2 0 29 Sep 2022
Towards Parameter-Efficient Integration of Pre-Trained Language Models In Temporal Video Grounding Erica K. Shimomoto Edison Marrese-Taylor Hiroya Takamura Ichiro Kobayashi Hideki Nakayama Yusuke Miyao 21 7 0 26 Sep 2022
Learning Visual Explanations for DCNN-Based Image Classifiers Using an Attention Mechanism Ioanna Gkartzonika Nikolaos Gkalelis Vasileios Mezaris 17 9 0 22 Sep 2022
How to Adapt Pre-trained Vision-and-Language Models to a Text-only Input? Lovisa Hagström Richard Johansson VLM 25 4 0 19 Sep 2022
Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions Paul Pu Liang Amir Zadeh Louis-Philippe Morency 16 60 0 07 Sep 2022
PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Model Tao Guo Song Guo Junxiao Wang Wenchao Xu FedML VLM LRM 16 110 0 24 Aug 2022
ILLUME: Rationalizing Vision-Language Models through Human Interactions Manuel Brack P. Schramowski Bjorn Deiseroth Kristian Kersting VLM MLLM 19 3 0 17 Aug 2022
Prompt Tuning for Generative Multimodal Pretrained Models Han Yang Junyang Lin An Yang Peng Wang Chang Zhou Hongxia Yang VLM LRM VPVLM 35 30 0 04 Aug 2022
Masked Vision and Language Modeling for Multi-modal Representation Learning Gukyeong Kwon Zhaowei Cai Avinash Ravichandran Erhan Bas Rahul Bhotika Stefano Soatto 22 67 0 03 Aug 2022
An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion Rinon Gal Yuval Alaluf Y. Atzmon Or Patashnik Amit H. Bermano Gal Chechik Daniel Cohen-Or 34 1,780 0 02 Aug 2022
Pro-tuning: Unified Prompt Tuning for Vision Tasks Xing Nie Bolin Ni Jianlong Chang Gaomeng Meng Chunlei Huo Zhaoxiang Zhang Shiming Xiang Qi Tian Chunhong Pan AAML VPVLM VLM 19 69 0 28 Jul 2022
LaKo: Knowledge-driven Visual Question Answering via Late Knowledge-to-Text Injection Zhuo Chen Yufen Huang Jiaoyan Chen Yuxia Geng Yin Fang Jeff Z. Pan Ningyu Zhang Wen Zhang 13 35 0 26 Jul 2022