A Concept-Based Explainability Framework for Large Multimodal Models

A Concept-Based Explainability Framework for Large Multimodal Models

12 June 2024

Papers citing "A Concept-Based Explainability Framework for Large Multimodal Models"

17 / 17 papers shown

Title
Interpreting the Linear Structure of Vision-language Model Embedding Spaces Isabel Papadimitriou Huangyuan Su Thomas Fel Naomi Saphra Sham Kakade Stephanie Gil VLM 40 0 0 16 Apr 2025
Sparse Autoencoders Learn Monosemantic Features in Vision-Language Models Mateusz Pach Shyamgopal Karthik Quentin Bouniot Serge Belongie Zeynep Akata VLM 62 0 0 03 Apr 2025
Re-Imagining Multimodal Instruction Tuning: A Representation View Yiyang Liu James Liang Ruixiang Tang Yugyung Lee Majid Rabbani ... Raghuveer M. Rao Lifu Huang Dongfang Liu Qifan Wang Cheng Han 63 0 0 02 Mar 2025
Causality Is Key to Understand and Balance Multiple Goals in Trustworthy ML and Foundation Models Ruta Binkyte Ivaxi Sheth Zhijing Jin Mohammad Havaei Bernhard Schölkopf Mario Fritz 70 0 0 28 Feb 2025
Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment Harrish Thasarathan Julian Forsyth Thomas Fel M. Kowal Konstantinos G. Derpanis 100 7 0 06 Feb 2025
Analyzing Fine-tuning Representation Shift for Multimodal LLMs Steering alignment Pegah Khayatan Mustafa Shukor Jayneel Parekh Matthieu Cord LLMSV 38 1 0 06 Jan 2025
Visual Large Language Models for Generalized and Specialized Applications Yifan Li Zhixin Lai Wentao Bao Zhen Tan Anh Dao Kewei Sui Jiayi Shen Dong Liu Huan Liu Yu Kong VLM 86 10 0 06 Jan 2025
Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey Yunkai Dang Kaichen Huang Jiahao Huo Yibo Yan S. Huang ... Kun Wang Yong Liu Jing Shao Hui Xiong Xuming Hu LRM 96 14 0 03 Dec 2024
Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens Zhangqi Jiang Junkai Chen Beier Zhu Tingjin Luo Yankun Shen Xu Yang 95 4 0 23 Nov 2024
MINER: Mining the Underlying Pattern of Modality-Specific Neurons in Multimodal Large Language Models Kaichen Huang Jiahao Huo Yibo Yan Kun Wang Yutao Yue Xuming Hu 31 2 0 07 Oct 2024
Concept-Based Explanations in Computer Vision: Where Are We and Where Could We Go? Jae Hee Lee Georgii Mikriukov Gesina Schwalbe Stefan Wermter D. Wolter 48 2 0 20 Sep 2024
Improved Baselines for Data-efficient Perceptual Augmentation of LLMs Théophane Vallaeys Mustafa Shukor Matthieu Cord Jakob Verbeek 54 12 0 20 Mar 2024
Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE) Usha Bhalla Alexander X. Oesterling Suraj Srinivas Flavio du Pin Calmon Himabindu Lakkaraju 34 35 0 16 Feb 2024
Attention Lens: A Tool for Mechanistically Interpreting the Attention Head Information Retrieval Mechanism Mansi Sakarvadia Arham Khan Aswathy Ajith Daniel Grzenda Nathaniel Hudson André Bauer Kyle Chard Ian T. Foster 172 9 0 25 Oct 2023
Referential communication in heterogeneous communities of pre-trained visual deep networks Matéo Mahaut Francesca Franzon Roberto Dessì Marco Baroni 14 7 0 04 Feb 2023
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models Junnan Li Dongxu Li Silvio Savarese Steven C. H. Hoi VLM MLLM 247 4,186 0 30 Jan 2023
Linearly Mapping from Image to Text Space Jack Merullo Louis Castricato Carsten Eickhoff Ellie Pavlick VLM 159 104 0 30 Sep 2022