v1v2 (latest)

Analyzing the Structure of Attention in a Transformer Language Model

7 June 2019

Papers citing "Analyzing the Structure of Attention in a Transformer Language Model"

50 / 226 papers shown

Title
Fisher Information-based Efficient Curriculum Federated Learning with Large Language ModelsConference on Empirical Methods in Natural Language Processing (EMNLP), 2024 Ji Liu Jiaxiang Ren Ruoming Jin Zijie Zhang Yang Zhou P. Valduriez Dejing Dou FedML 265 8 0 30 Sep 2024
Selective Prompt Anchoring for Code Generation Yuan Tian Tianyi Zhang 762 7 0 17 Aug 2024
Validating Mechanistic Interpretations: An Axiomatic Approach Nils Palumbo Ravi Mangal Zifan Wang Saranya Vijayakumar Corina S. Pasareanu Somesh Jha 305 1 0 18 Jul 2024
How and where does CLIP process negation? Vincent Quantmeyer Pablo Mosteiro Albert Gatt CoGe 238 11 0 15 Jul 2024
Beyond KV Caching: Shared Attention for Efficient LLMs Bingli Liao Danilo Vasconcellos Vargas 199 9 0 13 Jul 2024
Images Speak Louder than Words: Understanding and Mitigating Bias in Vision-Language Model from a Causal Mediation Perspective Zhaotian Weng Zijun Gao Jerone Andrews Jieyu Zhao 291 9 0 03 Jul 2024
ESALE: Enhancing Code-Summary Alignment Learning for Source Code Summarization Chunrong Fang Weisong Sun Yuchen Chen Xiao Chen Zhao Wei Quanjun Zhang Yudu You Bin Luo Yang Liu Zhenyu Chen AI4TS 297 20 0 01 Jul 2024
Confidence Regulation Neurons in Language Models Alessandro Stolfo Ben Wu Wes Gurnee Yonatan Belinkov Xingyi Song Mrinmaya Sachan Neel Nanda 232 39 0 24 Jun 2024
DeciMamba: Exploring the Length Extrapolation Potential of Mamba Assaf Ben-Kish Itamar Zimerman Shady Abu Hussein Nadav Cohen Amir Globerson Lior Wolf Raja Giryes Mamba 506 35 0 20 Jun 2024
A Primal-Dual Framework for Transformers and Neural Networks Tan M. Nguyen Tam Nguyen Nhat Ho Andrea L. Bertozzi Richard G. Baraniuk Stanley J. Osher ViT 183 16 0 19 Jun 2024
Elliptical Attention Stefan K. Nielsen Laziz U. Abdullaev R. Teo Tan M. Nguyen 278 5 0 19 Jun 2024
Unveiling the Hidden Structure of Self-Attention via Kernel Principal Component Analysis R. Teo Tan M. Nguyen 338 7 0 19 Jun 2024
Enhancing In-Context Learning Performance with just SVD-Based Weight Pruning: A Theoretical Perspective Xinhao Yao Xiaolin Hu Shenzhi Yang Yong Liu 240 3 0 06 Jun 2024
Understanding Retrieval Robustness for Retrieval-Augmented Image Captioning Wenyan Li Jiaang Li R. Ramos Raphael Tang Desmond Elliott VLM 290 9 0 04 Jun 2024
Language Models Need Inductive Biases to Count Inductively Yingshan Chang Yonatan Bisk LRM 260 14 0 30 May 2024
Self-Taught Recognizer: Toward Unsupervised Adaptation for Speech Foundation ModelsNeural Information Processing Systems (NeurIPS), 2024 Yuchen Hu Chen Chen Chao-Han Huck Yang Chengwei Qin Pin-Yu Chen Chng Eng Siong Chao Zhang VLM 191 7 0 23 May 2024
A Masked Semi-Supervised Learning Approach for Otago Micro Labels Recognition Meng Shang L. Dedeyne J. Dupont Laura Vercauteren Nadjia Amini ... E. Gielen Sabine Verschueren Carolina Varon W. de Raedt Bart Vanrumste 237 0 0 21 May 2024
How does Multi-Task Training Affect Transformer In-Context Capabilities? Investigations with Function ClassesNorth American Chapter of the Association for Computational Linguistics (NAACL), 2024 Harmon Bhasin Timothy Ossowski Yiqiao Zhong Junjie Hu 103 2 0 04 Apr 2024
Using Interpretation Methods for Model EnhancementConference on Empirical Methods in Natural Language Processing (EMNLP), 2024 Zhuo Chen Chengyue Jiang Kewei Tu 289 3 0 02 Apr 2024
Unveiling and Mitigating Memorization in Text-to-image Diffusion Models through Cross Attention Jie Ren Yaxin Li Shenglai Zeng Han Xu Lingjuan Lyu Yue Xing Shucheng Zhou 376 46 0 17 Mar 2024
Are Human Conversations Special? A Large Language Model Perspective Toshish Jawale Chaitanya Animesh Sekhar Vallath Kartik Talamadupula Larry Heck 189 3 0 08 Mar 2024
On the Challenges and Opportunities in Generative AI Laura Manduchi Kushagra Pandey Kushagra Pandey Robert Bamler Sina Daubener ... Yixin Wang F. Wenzel Frank Wood Stephan Mandt Vincent Fortuin 732 40 0 28 Feb 2024
What Do Language Models Hear? Probing for Auditory Representations in Language Models Jerry Ngo Yoon Kim AuLLM MILM 178 13 0 26 Feb 2024
How Large Language Models Encode Context Knowledge? A Layer-Wise Probing Study Tianjie Ju Weiwei Sun Wei Du Xinwei Yuan Zhaochun Ren Gongshen Liu KELM 198 55 0 25 Feb 2024
CAMELoT: Towards Large Language Models with Training-Free Consolidated Associative Memory Zexue He Leonid Karlinsky Donghyun Kim Julian McAuley Dmitry Krotov Rogerio Feris KELM RALM 190 19 0 21 Feb 2024
A Mechanistic Analysis of a Transformer Trained on a Symbolic Multi-Step Reasoning Task Jannik Brinkmann Abhay Sheshadri Victor Levoso Paul Swoboda Christian Bartelt LRM 248 46 0 19 Feb 2024
Attention with Markov: A Framework for Principled Analysis of Transformers via Markov Chains Ashok Vardhan Makkuva Marco Bondaschi Adway Girish Alliot Nagle Martin Jaggi Hyeji Kim Michael C. Gastpar OffRL 374 37 0 06 Feb 2024
Attention Meets Post-hoc Interpretability: A Mathematical PerspectiveInternational Conference on Machine Learning (ICML), 2024 Gianluigi Lopardo F. Precioso Damien Garreau 249 11 0 05 Feb 2024
Transformer-Based Models Are Not Yet Perfect At Learning to Emulate Structural Recursion Dylan Zhang Curt Tigges Zory Zhang Stella Biderman Maxim Raginsky Talia Ringer 236 19 0 23 Jan 2024
Pruning for Protection: Increasing Jailbreak Resistance in Aligned LLMs Without Fine-Tuning Adib Hasan Ileana Rugina Alex Wang AAML 232 29 0 19 Jan 2024
Model Editing Harms General Abilities of Large Language Models: Regularization to the RescueConference on Empirical Methods in Natural Language Processing (EMNLP), 2024 Jia-Chen Gu Haoyang Xu Jun-Yu Ma Pan Lu Zhen-Hua Ling Kai-Wei Chang Nanyun Peng KELM 450 81 0 09 Jan 2024
Modeling Uncertainty in Personalized Emotion Prediction with Normalizing Flows P. Milkowski Konrad Karanowski Patryk Wielopolski Jan Kocoñ Przemyslaw Kazienko Maciej Zikeba 207 7 0 10 Dec 2023
Transformers are uninterpretable with myopic methods: a case study with bounded Dyck grammarsNeural Information Processing Systems (NeurIPS), 2023 Kaiyue Wen Yuchen Li Bing Liu Andrej Risteski 278 27 0 03 Dec 2023
Mitigating Over-smoothing in Transformers via Regularized Nonlocal FunctionalsNeural Information Processing Systems (NeurIPS), 2023 Tam Nguyen Tan-Minh Nguyen Richard G. Baraniuk 174 25 0 01 Dec 2023
Linear Log-Normal Attention with Unbiased ConcentrationInternational Conference on Learning Representations (ICLR), 2023 Yury Nahshan Dor-Joseph Kampeas E. Haleva 266 9 0 22 Nov 2023
Visual Analytics for Generative Transformer Models Raymond Li Ruixin Yang Wen Xiao Ahmed AbuRaed Gabriel Murray Giuseppe Carenini 205 3 0 21 Nov 2023
On the Importance of Step-wise Embeddings for Heterogeneous Clinical Time-Series Rita Kuznetsova Alizée Pace Manuel Burger Hugo Yèche Gunnar Rätsch AI4TS 219 6 0 15 Nov 2023
Legal-HNet: Mixing Legal Long-Context Tokens with Hartley Transform Daniele Giofré Sneha Ghantasala AILaw 138 0 0 09 Nov 2023
Uncovering Intermediate Variables in Transformers using Circuit Probing Michael A. Lepori Thomas Serre Ellie Pavlick 384 11 0 07 Nov 2023
p-Laplacian Transformer Tuan Nguyen Tam Nguyen Vinh-Tiep Nguyen Tan-Minh Nguyen 174 0 0 06 Nov 2023
Increasing The Performance of Cognitively Inspired Data-Efficient Language Models via Implicit Structure Building Omar Momen David Arps Laura Kallmeyer AI4CE 194 4 0 31 Oct 2023
Roles of Scaling and Instruction Tuning in Language Perception: Model vs. Human AttentionConference on Empirical Methods in Natural Language Processing (EMNLP), 2023 Changjiang Gao Shujian Huang Jixing Li Jiajun Chen LRM ALM 331 8 0 29 Oct 2023
Towards a Mechanistic Interpretation of Multi-Step Reasoning Capabilities of Language ModelsConference on Empirical Methods in Natural Language Processing (EMNLP), 2023 Buse Giledereli Jiaoda Li Yu Fei Alessandro Stolfo Wangchunshu Zhou Guangtao Zeng Antoine Bosselut Mrinmaya Sachan LRM 377 59 0 23 Oct 2023
REFER: An End-to-end Rationale Extraction Framework for Explanation RegularizationConference on Computational Natural Language Learning (CoNLL), 2023 Mohammad Reza Ghasemi Madani Pasquale Minervini 227 4 0 22 Oct 2023
Plausibility Processing in Transformer Language Models: Focusing on the Role of Attention Heads in GPT Soo Hyun Ryu 146 0 0 20 Oct 2023
Disentangling the Linguistic Competence of Privacy-Preserving BERTBlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP), 2023 Stefan Arnold Nils Kemmerzell Annika Schreiner 248 0 0 17 Oct 2023
Untying the Reversal Curse via Bidirectional Language Model Editing Jun-Yu Ma Jia-Chen Gu Zhen-Hua Ling Quan Liu Cong Liu KELM 286 41 0 16 Oct 2023
Interpreting and Exploiting Functional Specialization in Multi-Head Attention under Multi-task LearningConference on Empirical Methods in Natural Language Processing (EMNLP), 2023 Chong Li Shaonan Wang Yunhao Zhang Jiajun Zhang Chengqing Zong 208 7 0 16 Oct 2023
Rethinking Model Selection and Decoding for Keyphrase Generation with Pre-trained Sequence-to-Sequence ModelsConference on Empirical Methods in Natural Language Processing (EMNLP), 2023 Di Wu Wasi Uddin Ahmad Kai-Wei Chang 239 12 0 10 Oct 2023
Breaking Down Word Semantics from Pre-trained Language Models through Layer-wise Dimension Selection Nayoung Choi 82 0 0 08 Oct 2023