Multimodal Neurons in Pretrained Text-Only Transformers

3 August 2023

Antonio Torralba

Papers citing "Multimodal Neurons in Pretrained Text-Only Transformers"

26 / 26 papers shown

Title
Advancing Multimodal In-Context Learning in Large Vision-Language Models with Task-aware Demonstrations Yanshu Li 44 0 0 05 Mar 2025
Analyzing Fine-tuning Representation Shift for Multimodal LLMs Steering alignment Pegah Khayatan Mustafa Shukor Jayneel Parekh Matthieu Cord LLMSV 38 1 0 06 Jan 2025
Visual Large Language Models for Generalized and Specialized Applications Yifan Li Zhixin Lai Wentao Bao Zhen Tan Anh Dao Kewei Sui Jiayi Shen Dong Liu Huan Liu Yu Kong VLM 86 11 0 06 Jan 2025
Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey Yunkai Dang Kaichen Huang Jiahao Huo Yibo Yan S. Huang ... Kun Wang Yong Liu Jing Shao Hui Xiong Xuming Hu LRM 96 14 0 03 Dec 2024
Towards Interpreting Visual Information Processing in Vision-Language Models Clement Neo Luke Ong Philip H. S. Torr Mor Geva David M. Krueger Fazl Barez 84 6 0 09 Oct 2024
MINER: Mining the Underlying Pattern of Modality-Specific Neurons in Multimodal Large Language Models Kaichen Huang Jiahao Huo Yibo Yan Kun Wang Yutao Yue Xuming Hu 31 2 0 07 Oct 2024
Neuron-Level Sequential Editing for Large Language Models Houcheng Jiang Junfeng Fang Tianyu Zhang An Zhang Ruipeng Wang Tao Liang Xiang Wang KELM 43 4 0 05 Oct 2024
Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations Nick Jiang Anish Kachinthaya Suzie Petryk Yossi Gandelsman VLM 32 14 0 03 Oct 2024
Attribution Analysis Meets Model Editing: Advancing Knowledge Correction in Vision Language Models with VisEdit Qizhou Chen Taolin Zhang Chengyu Wang Xiaofeng He Dakan Wang Tingting Liu KELM 44 2 0 19 Aug 2024
Knowledge Mechanisms in Large Language Models: A Survey and Perspective Meng Wang Yunzhi Yao Ziwen Xu Shuofei Qiao Shumin Deng ... Yong-jia Jiang Pengjun Xie Fei Huang Huajun Chen Ningyu Zhang 47 28 0 22 Jul 2024
MMNeuron: Discovering Neuron-Level Domain-Specific Interpretation in Multimodal Large Language Model Jiahao Huo Yibo Yan Boren Hu Yutao Yue Xuming Hu LRM MLLM 32 7 0 17 Jun 2024
Concept-skill Transferability-based Data Selection for Large Vision-Language Models Jaewoo Lee Boyang Li Sung Ju Hwang VLM 33 8 0 16 Jun 2024
A Concept-Based Explainability Framework for Large Multimodal Models Jayneel Parekh Pegah Khayatan Mustafa Shukor A. Newson Matthieu Cord 32 16 0 12 Jun 2024
Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs Mustafa Shukor Matthieu Cord 64 5 0 26 May 2024
Automatic Discovery of Visual Circuits Achyuta Rajaram Neil Chowdhury Antonio Torralba Jacob Andreas Sarah Schwettmann GNN 19 3 0 22 Apr 2024
Mysterious Projections: Multimodal LLMs Gain Domain-Specific Visual Capabilities Without Richer Cross-Modal Projections Gaurav Verma Minje Choi Kartik Sharma J. Watson-Daniels Sejoon Oh Srijan Kumar MLLM VLM 32 8 0 26 Feb 2024
Visual In-Context Learning for Large Vision-Language Models Yucheng Zhou Xiang Li Qianning Wang Jianbing Shen MLLM 27 57 0 18 Feb 2024
Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models Asma Ghandeharioun Avi Caciularu Adam Pearce Lucas Dixon Mor Geva 25 87 0 11 Jan 2024
Multimodal Attention Merging for Improved Speech Recognition and Audio Event Classification Anirudh S. Sundar Chao-Han Huck Yang David M. Chan Shalini Ghosh Venkatesh Ravichandran P. S. Nidadavolu MoMe 38 8 0 22 Dec 2023
Object Recognition as Next Token Prediction Kaiyu Yue Borchun Chen Jonas Geiping Hengduo Li Tom Goldstein Ser-Nam Lim 22 8 0 04 Dec 2023
Representing visual classification as a linear combination of words Shobhit Agarwal Yevgeniy R. Semenov William Lotter 25 1 0 18 Nov 2023
Finding and Editing Multi-Modal Neurons in Pre-Trained Transformers Haowen Pan Yixin Cao Xiaozhi Wang Xun Yang Meng Wang KELM 36 24 0 13 Nov 2023
Frozen Transformers in Language Models Are Effective Visual Encoder Layers Ziqi Pang Ziyang Xie Yunze Man Yu-xiong Wang 40 25 0 19 Oct 2023
Linearly Mapping from Image to Text Space Jack Merullo Louis Castricato Carsten Eickhoff Ellie Pavlick VLM 159 104 0 30 Sep 2022
Natural Language Descriptions of Deep Visual Features Evan Hernandez Sarah Schwettmann David Bau Teona Bagashvili Antonio Torralba Jacob Andreas MILM 196 116 0 26 Jan 2022
The Power of Scale for Parameter-Efficient Prompt Tuning Brian Lester Rami Al-Rfou Noah Constant VPVLM 280 3,843 0 18 Apr 2021