v1v2 (latest)

Analyzing the Structure of Attention in a Transformer Language Model

7 June 2019

Papers citing "Analyzing the Structure of Attention in a Transformer Language Model"

50 / 225 papers shown

Title
Order-Level Attention Similarity Across Language Models: A Latent Commonality Jinglin Liang Jin Zhong Shuangping Huang Yunqing Hu Huiyuan Zhang Huifang Li Lixin Fan Hanlin Gu 64 0 0 07 Nov 2025
CircuitSeer: Mining High-Quality Data by Probing Mathematical Reasoning Circuits in LLMs Shaobo Wang Yongliang Miao Yuancheng Liu Qianli Ma Ning Liao Linfeng Zhang LRM 105 1 0 21 Oct 2025
CAST: Compositional Analysis via Spectral Tracking for Understanding Transformer Layer Functions Zihao Fu Ming Liao Chris Russell Zhenguang G. Cai 68 0 0 16 Oct 2025
Algorithmic Primitives and Compositional Geometry of Reasoning in Language Models Samuel Lippl Thomas McGee Kimberly Lopez Ziwen Pan Pierce Zhang Salma Ziadi Oliver Eberle Ida Momennejad LRM 86 0 0 13 Oct 2025
CacheClip: Accelerating RAG with Effective KV Cache Reuse Bin Yang Qiuyu Leng Jun Zeng Zhenhua Wu VLM 48 0 0 11 Oct 2025
There is More to Attention: Statistical Filtering Enhances Explanations in Vision Transformers Meghna P. Ayyar Jenny Benois-Pineau A. Zemmari 76 1 0 07 Oct 2025
AIMCoT: Active Information-driven Multimodal Chain-of-Thought for Vision-Language Reasoning Xiping Li Jianghong Ma LRM 49 0 0 30 Sep 2025
Task Vectors, Learned Not Extracted: Performance Gains and Mechanistic Insight Haolin Yang Hakaze Cho Kaize Ding Naoya Inoue 96 0 0 29 Sep 2025
On the Capacity of Self-Attention Micah Adler 129 0 0 26 Sep 2025
From Input Perception to Predictive Insight: Modeling Model Blind Spots Before They Become Errors Maggie Mi Aline Villavicencio Nafise Sadat Moosavi 68 0 0 24 Sep 2025
Uncovering Graph Reasoning in Decoder-only Transformers with Circuit Tracing Xinnan Dai Chung-Hsiang Lo Kai Guo Shenglai Zeng Dongsheng Luo Shucheng Zhou 65 1 0 24 Sep 2025
Cross-Attention is Half Explanation in Speech-to-Text Models Sara Papi Dennis Fucci Marco Gaido Matteo Negri L. Bentivogli LRM 100 0 0 22 Sep 2025
Steering Language Models in Multi-Token Generation: A Case Study on Tense and Aspect Alina Klerings Jannik Brinkmann Daniel Ruffinelli Simone Paolo Ponzetto LLMSV 94 0 0 15 Sep 2025
NeuroBreak: Unveil Internal Jailbreak Mechanisms in Large Language Models Chuhan Zhang Ye Zhang Bowen Shi Yuyou Gan Xuhong Zhang S. Ji Dazhan Deng Yingcai Wu AAML 84 0 0 04 Sep 2025
MindGuard: Tracking, Detecting, and Attributing MCP Tool Poisoning Attack via Decision Dependence Graph Zhiqiang Wang Junyang Zhang Guanquan Shi Haoran Cheng Yunhao Yao Kaiwen Guo Haohua Du Xiang-Yang Li 72 5 0 28 Aug 2025
The Cultural Gene of Large Language Models: A Study on the Impact of Cross-Corpus Training on Model Values and Biases Emanuel Z. Fenech-Borg Tilen P. Meznaric-Kos Milica D. Lekovic-Bojovic Arni J. Hentze-Djurhuus 100 0 0 17 Aug 2025
On the Risk of Misleading Reports: Diagnosing Textual Biases in Multimodal Clinical AI David Restrepo Ira Ktena Maria Vakalopoulou Stergios Christodoulidis Enzo Ferrante 76 0 0 31 Jul 2025
Wavelet Logic Machines: Learning and Reasoning in the Spectral Domain Without Neural Networks Andrew Kiruluta OOD AI4TS 65 0 0 18 Jul 2025
Overcoming Long-Context Limitations of State-Space Models via Context-Dependent Sparse Attention Zhihao Zhan Jianan Zhao Zhaocheng Zhu Jian Tang 159 1 0 01 Jul 2025
Through the Stealth Lens: Rethinking Attacks and Defenses in RAG Sarthak Choudhary Nils Palumbo Ashish Hooda Krishnamurthy Dvijotham Somesh Jha 160 2 0 04 Jun 2025
Multi-Scale Manifold Alignment for Interpreting Large Language Models: A Unified Information-Geometric Framework Yukun Zhang Qi Dong 101 0 0 24 May 2025
Exploring Implicit Visual Misunderstandings in Multimodal Large Language Models through Attention Analysis Pengfei Wang Guohai Xu Weinong Wang Junjie Yang Jie Lou Yunhua Xue 266 1 0 15 May 2025
Weight-of-Thought Reasoning: Exploring Neural Network Weights for Enhanced LLM Reasoning Saif Punjwani Larry Heck LRM 183 0 0 14 Apr 2025
Follow the Flow: On Information Flow Across Textual Tokens in Text-to-Image Models Guy Kaplan Michael Toker Yuval Reif Yonatan Belinkov Roy Schwartz DiffM 307 2 0 01 Apr 2025
Are formal and functional linguistic mechanisms dissociated in language models? Michael Hanna Sandro Pezzelle Yonatan Belinkov 413 4 0 14 Mar 2025
AxBERT: An Interpretable Chinese Spelling Correction Method Driven by Associative Knowledge Network Fanyu Wang Hangyu Zhu Zhenping Xie 173 0 0 04 Mar 2025
Transformer Meets Twicing: Harnessing Unattended Residual InformationInternational Conference on Learning Representations (ICLR), 2025 Laziz U. Abdullaev Tan M. Nguyen 380 4 0 02 Mar 2025
Retrieval Backward Attention without Additional Training: Enhance Embeddings of Large Language Models via Repetition Yifei Duan Raphael Shang Deng Liang Yongqiang Cai 275 0 0 28 Feb 2025
Steered Generation via Gradient Descent on Sparse Features Sumanta Bhattacharyya Pedram Rooshenas LLMSV 216 0 0 25 Feb 2025
On the Robustness of Transformers against Context Hijacking for Linear Classification Tianle Li Chenyang Zhang Xingwu Chen Yuan Cao Difan Zou 309 3 0 24 Feb 2025
What are Models Thinking about? Understanding Large Language Model Hallucinations "Psychology" through Model Inner State Analysis Peiran Wang Yang Liu Yunfei Lu Jue Hong Ye Wu HILM LRM 202 1 0 20 Feb 2025
PTQ1.61: Push the Real Limit of Extremely Low-Bit Post-Training Quantization Methods for Large Language ModelsAnnual Meeting of the Association for Computational Linguistics (ACL), 2025 Jiaqi Zhao Miao Zhang Ming Wang Yuzhang Shang Kaihao Zhang Weili Guan Yaowei Wang Min Zhang MQ 290 2 0 18 Feb 2025
MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connections Da Xiao Qingye Meng Shengping Li Xingyuan Yuan MoE AI4CE 396 7 0 13 Feb 2025
Top-Theta Attention: Sparsifying Transformers by Compensated Thresholding Konstantin Berestizshevsky Renzo Andri Lukas Cavigelli 309 2 0 12 Feb 2025
Padding Tone: A Mechanistic Analysis of Padding Tokens in T2I ModelsNorth American Chapter of the Association for Computational Linguistics (NAACL), 2025 Michael Toker Ido Galil Hadas Orgad Rinon Gal Yoad Tewel Gal Chechik Yonatan Belinkov DiffM 187 5 0 12 Jan 2025
Dynamic Attention-Guided Context Decoding for Mitigating Context Faithfulness Hallucinations in Large Language ModelsAnnual Meeting of the Association for Computational Linguistics (ACL), 2025 Yanwen Huang Yong Zhang Ning Cheng Zhitao Li Shaojun Wang Jing Xiao 342 4 0 02 Jan 2025
Domain-adaptative Continual Learning for Low-resource Tasks: Evaluation on Nepali Sharad Duwal Suraj Prasai Suresh Manandhar CLL 187 3 0 18 Dec 2024
Analyzing the Attention Heads for Pronoun Disambiguation in Context-aware Machine Translation Models Paweł Mąka Yusuf Can Semerci Jan Scholtes Gerasimos Spanakis 209 1 0 15 Dec 2024
SEEKR: Selective Attention-Guided Knowledge Retention for Continual Learning of Large Language ModelsConference on Empirical Methods in Natural Language Processing (EMNLP), 2024 Jinghan He Haiyun Guo Kuan Zhu Zihan Zhao Ming Tang Jinqiao Wang KELM 246 8 0 09 Nov 2024
Causal Interventions on Causal Paths: Mapping GPT-2's Reasoning From Syntax to Semantics Isabelle Lee Joshua Lum Ziyi Liu Dani Yogatama LRM 117 0 0 28 Oct 2024
From Imitation to Introspection: Probing Self-Consciousness in Language ModelsAnnual Meeting of the Association for Computational Linguistics (ACL), 2024 Sirui Chen Shu Yu Shengjie Zhao Chaochao Lu MILM LRM 344 8 0 24 Oct 2024
On Explaining with Attention MatricesEuropean Conference on Artificial Intelligence (ECAI), 2024 Omar Naim Nicholas Asher 156 3 0 24 Oct 2024
A Psycholinguistic Evaluation of Language Models' Sensitivity to Argument RolesConference on Empirical Methods in Natural Language Processing (EMNLP), 2024 Eun-Kyoung Rosa Lee Sathvik Nair Naomi Feldman 239 5 0 21 Oct 2024
AERO: Entropy-Guided Framework for Private LLM Inference N. Jha Brandon Reagen 341 5 0 16 Oct 2024
Generative AI's aggregated knowledge versus web-based curated knowledge Ted Selker Yunzi Wu 74 0 0 15 Oct 2024
Impacts of Continued Legal Pre-Training and IFT on LLMs' Latent Representations of Human-Defined Legal ConceptsInternational Conference on Legal Knowledge and Information Systems (JURIX), 2024 Shaun Ho AILaw 197 0 0 15 Oct 2024
ReLU's Revival: On the Entropic Overload in Normalization-Free Large Language Models N. Jha Brandon Reagen OffRL AI4CE 289 3 0 12 Oct 2024
Power-Softmax: Towards Secure LLM Inference over Encrypted Data Itamar Zimerman Allon Adir E. Aharoni Matan Avitan Moran Baruch Nir Drucker Jenny Lerner Ramy Masalha Reut Meiri Omri Soceanu 144 7 0 12 Oct 2024
Duo-LLM: A Framework for Studying Adaptive Computation in Large Language Models Keivan Alizadeh Iman Mirzadeh Hooman Shahrokhi Dmitry Belenko Frank Sun Minsik Cho Mohammad Hossein Sekhavat Moin Nabi Mehrdad Farajtabar MoE 205 2 0 01 Oct 2024
Fisher Information-based Efficient Curriculum Federated Learning with Large Language ModelsConference on Empirical Methods in Natural Language Processing (EMNLP), 2024 Ji Liu Jiaxiang Ren Ruoming Jin Zijie Zhang Yang Zhou P. Valduriez Dejing Dou FedML 237 8 0 30 Sep 2024