Transformer Feed-Forward Layers Are Key-Value Memories

29 December 2020

Papers citing "Transformer Feed-Forward Layers Are Key-Value Memories"

50 / 128 papers shown

Title
Pre-training vs. Fine-tuning: A Reproducibility Study on Dense Retrieval Knowledge Acquisition Zheng Yao Shuai Wang Guido Zuccon 21 0 0 12 May 2025
Attention Is Not All You Need: The Importance of Feedforward Networks in Transformer Models Isaac Gerber 29 0 0 10 May 2025
Short-circuiting Shortcuts: Mechanistic Investigation of Shortcuts in Text Classification Leon Eshuijs Shihan Wang Antske Fokkens 26 0 0 09 May 2025
FloE: On-the-Fly MoE Inference on Memory-constrained GPU Yuxin Zhou Zheng Li J. Zhang Jue Wang Y. Wang Zhongle Xie Ke Chen Lidan Shou MoE 46 0 0 09 May 2025
Demystifying optimized prompts in language models Rimon Melamed Lucas H. McCabe H. H. Huang 39 0 0 04 May 2025
Unlearning Sensitive Information in Multimodal LLMs: Benchmark and Attack-Defense Evaluation Vaidehi Patil Yi-Lin Sung Peter Hase Jie Peng Tianlong Chen Mohit Bansal AAML MU 81 3 0 01 May 2025
On the generalization of language models from in-context learning and finetuning: a controlled study Andrew Kyle Lampinen Arslan Chaudhry Stephanie Chan Cody Wild Diane Wan Alex Ku Jorg Bornschein Razvan Pascanu Murray Shanahan James L. McClelland 46 0 0 01 May 2025
SetKE: Knowledge Editing for Knowledge Elements Overlap Yifan Wei Xiaoyan Yu Ran Song Hao Peng Angsheng Li KELM 53 0 0 29 Apr 2025
R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference Zhenyu (Allen) Zhang Zechun Liu Yuandong Tian Harshit Khaitan Z. Wang Steven Li 57 0 0 28 Apr 2025
Revisiting Transformers through the Lens of Low Entropy and Dynamic Sparsity Ruifeng Ren Yong Liu 111 0 0 26 Apr 2025
Param $Δ$ for Direct Weight Mixing: Post-Train Large Language Model at Zero Cost Sheng Cao Mingrui Wu Karthik Prasad Yuandong Tian Zechun Liu MoMe 80 0 0 23 Apr 2025
Bigram Subnetworks: Mapping to Next Tokens in Transformer Language Models Tyler A. Chang Benjamin Bergen 48 0 0 21 Apr 2025
Model Utility Law: Evaluating LLMs beyond Performance through Mechanism Interpretable Metric Yixin Cao Jiahao Ying Y. Wang Xipeng Qiu Xuanjing Huang Yugang Jiang ELM 33 2 0 10 Apr 2025
Effective Skill Unlearning through Intervention and Abstention Yongce Li Chung-En Sun Tsui-Wei Weng MU 137 0 0 27 Mar 2025
Are formal and functional linguistic mechanisms dissociated in language models? Michael Hanna Sandro Pezzelle Yonatan Belinkov 45 0 0 14 Mar 2025
BioMamba: Leveraging Spectro-Temporal Embedding in Bidirectional Mamba for Enhanced Biosignal Classification Jian Qian Teck Lun Goh Bingyu Xie Chengyao Zhu Biao Wan Yawen Guan Rachel Ding Chen Patrick Chiang Mamba 45 0 0 14 Mar 2025
Discovering Influential Neuron Path in Vision Transformers Yifan Wang Yifei Liu Yingdong Shi C. Li Anqi Pang Sibei Yang Jingyi Yu Kan Ren ViT 69 0 0 12 Mar 2025
Exploiting Edited Large Language Models as General Scientific Optimizers Qitan Lv T. Liu H. Wang 36 0 0 08 Mar 2025
Strategy Coopetition Explains the Emergence and Transience of In-Context Learning Aaditya K. Singh Ted Moskovitz Sara Dragutinovic Felix Hill Stephanie C. Y. Chan Andrew Saxe 127 0 0 07 Mar 2025
SePer: Measure Retrieval Utility Through The Lens Of Semantic Perplexity Reduction Lu Dai Yijie Xu Jinhui Ye Hao Liu Hui Xiong 3DV RALM 78 2 0 03 Mar 2025
Triple Phase Transitions: Understanding the Learning Dynamics of Large Language Models from a Neuroscience Perspective Yuko Nakagi Keigo Tada Sota Yoshino Shinji Nishimoto Yu Takagi LRM 37 0 0 28 Feb 2025
Finite State Automata Inside Transformers with Chain-of-Thought: A Mechanistic Study on State Tracking Yifan Zhang Wenyu Du Dongming Jin Jie Fu Zhi Jin LRM 46 0 0 27 Feb 2025
Proactive Privacy Amnesia for Large Language Models: Safeguarding PII with Negligible Impact on Model Utility Martin Kuo Jingyang Zhang Jianyi Zhang Minxue Tang Louis DiValentin ... William Chen Amin Hass Tianlong Chen Y. Chen H. Li MU KELM 37 2 0 24 Feb 2025
Revealing and Mitigating Over-Attention in Knowledge Editing Pinzheng Wang Zecheng Tang Keyan Zhou J. Li Qiaoming Zhu M. Zhang KELM 115 2 0 21 Feb 2025
Repetition Neurons: How Do Language Models Produce Repetitions? Tatsuya Hiraoka Kentaro Inui MILM 73 6 0 21 Feb 2025
An LLM-based Agent for Reliable Docker Environment Configuration Ruida Hu Chao Peng Xinchen Wang Cuiyun Gao 65 2 0 19 Feb 2025
The Knowledge Microscope: Features as Better Analytical Lenses than Neurons Yuheng Chen Pengfei Cao Kang Liu Jun Zhao 43 0 0 18 Feb 2025
DSMoE: Matrix-Partitioned Experts with Dynamic Routing for Computation-Efficient Dense LLMs Minxuan Lv Zhenpeng Su Leiyu Pan Yizhe Xiong Zijia Lin ... Guiguang Ding Cheng Luo Di Zhang Kun Gai Songlin Hu MoE 39 0 0 18 Feb 2025
Mechanistic Unveiling of Transformer Circuits: Self-Influence as a Key to Model Reasoning L. Zhang Lijie Hu Di Wang LRM 88 0 0 17 Feb 2025
DELMAN: Dynamic Defense Against Large Language Model Jailbreaking with Model Editing Yi Wang Fenghua Weng S. Yang Zhan Qin Minlie Huang Wenjie Wang KELM AAML 48 0 0 17 Feb 2025
AnyEdit: Edit Any Knowledge Encoded in Language Models Houcheng Jiang Junfeng Fang Ningyu Zhang Guojun Ma Mingyang Wan X. Wang Xiangnan He Tat-Seng Chua KELM 49 8 0 08 Feb 2025
Understanding and Mitigating Gender Bias in LLMs via Interpretable Neuron Editing Zeping Yu Sophia Ananiadou KELM 43 1 0 24 Jan 2025
Transforming Vision Transformer: Towards Efficient Multi-Task Asynchronous Learning Hanwen Zhong Jiaxin Chen Yutong Zhang Di Huang Yunhong Wang MoE 42 0 0 12 Jan 2025
Key-value memory in the brain Samuel J. Gershman Ila Fiete Kazuki Irie 34 7 0 06 Jan 2025
Decoupling Knowledge and Reasoning in Transformers: A Modular Architecture with Generalized Cross-Attention Zhenyu Guo Wenguang Chen 37 0 0 01 Jan 2025
Out-of-distribution generalization via composition: a lens through induction heads in Transformers Jiajun Song Zhuoyan Xu Yiqiao Zhong 80 4 0 31 Dec 2024
How Do Artificial Intelligences Think? The Three Mathematico-Cognitive Factors of Categorical Segmentation Operated by Synthetic Neurons Michael Pichat William Pogrund Armanush Gasparian Paloma Pichat Samuel Demarchi Michael Veillet-Guillem 42 3 0 26 Dec 2024
Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projection Le Yang Ziwei Zheng Boxu Chen Zhengyu Zhao Chenhao Lin Chao Shen VLM 138 3 0 18 Dec 2024
Bridging the Visual Gap: Fine-Tuning Multimodal Models with Knowledge-Adapted Captions Moran Yanuka Assaf Ben-Kish Yonatan Bitton Idan Szpektor Raja Giryes VLM 45 2 0 13 Nov 2024
Controllable Context Sensitivity and the Knob Behind It Julian Minder Kevin Du Niklas Stoehr Giovanni Monea Chris Wendler Robert West Ryan Cotterell KELM 44 3 0 11 Nov 2024
TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters Haiyang Wang Yue Fan Muhammad Ferjad Naeem Yongqin Xian J. E. Lenssen Liwei Wang F. Tombari Bernt Schiele 41 2 0 30 Oct 2024
WAGLE: Strategic Weight Attribution for Effective and Modular Unlearning in Large Language Models Jinghan Jia Jiancheng Liu Yihua Zhang Parikshit Ram Nathalie Baracaldo Sijia Liu MU 35 2 0 23 Oct 2024
Steering Knowledge Selection Behaviours in LLMs via SAE-Based Representation Engineering Yu Zhao Alessio Devoto Giwon Hong Xiaotang Du Aryo Pradipta Gema Hongru Wang Xuanli He Kam-Fai Wong Pasquale Minervini KELM LLMSV 32 16 0 21 Oct 2024
ReDeEP: Detecting Hallucination in Retrieval-Augmented Generation via Mechanistic Interpretability ZhongXiang Sun Xiaoxue Zang Kai Zheng Yang Song Jun Xu Xiao Zhang Weijie Yu Yang Song Han Li 55 7 0 15 Oct 2024
Bypassing the Exponential Dependency: Looped Transformers Efficiently Learn In-context by Multi-step Gradient Descent Bo Chen Xiaoyu Li Yingyu Liang Zhenmei Shi Zhao-quan Song 86 19 0 15 Oct 2024
Dissecting Fine-Tuning Unlearning in Large Language Models Yihuai Hong Yuelin Zou Lijie Hu Ziqian Zeng Di Wang Haiqin Yang AAML MU 37 2 0 09 Oct 2024
Temporal Reasoning Transfer from Text to Video Lei Li Yuanxin Liu Linli Yao Peiyuan Zhang Chenxin An Lean Wang Xu Sun Lingpeng Kong Qi Liu LRM 40 7 0 08 Oct 2024
From Tokens to Words: On the Inner Lexicon of LLMs Guy Kaplan Matanel Oren Yuval Reif Roy Schwartz 46 12 0 08 Oct 2024
A Probabilistic Perspective on Unlearning and Alignment for Large Language Models Yan Scholten Stephan Günnemann Leo Schwinn MU 55 6 0 04 Oct 2024
Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Models Xin Zou Yizhou Wang Yibo Yan Yuanhuiyi Lyu Kening Zheng ... Junkai Chen Peijie Jiang J. Liu Chang Tang Xuming Hu 81 7 0 04 Oct 2024