Transformer Normalisation Layers and the Independence of Semantic Subspaces

25 June 2024

Papers citing "Transformer Normalisation Layers and the Independence of Semantic Subspaces"

13 / 13 papers shown

Title
HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization Zhijian Zhuo Yutao Zeng Ya Wang Sijun Zhang Jian Yang Xiaoqing Li Xun Zhou Jinwen Ma 46 0 0 06 Mar 2025
What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formation Aaditya K. Singh Ted Moskovitz Felix Hill Stephanie C. Y. Chan Andrew M. Saxe AI4CE 42 24 0 10 Apr 2024
On the Origins of Linear Representations in Large Language Models Yibo Jiang Goutham Rajendran Pradeep Ravikumar Bryon Aragam Victor Veitch 59 24 0 06 Mar 2024
Information Flow Routes: Automatically Interpreting Language Models at Scale Javier Ferrando Elena Voita 40 34 0 27 Feb 2024
The Transient Nature of Emergent In-Context Learning in Transformers Aaditya K. Singh Stephanie C. Y. Chan Ted Moskovitz Erin Grant Andrew M. Saxe Felix Hill 62 31 0 14 Nov 2023
On the Expressivity Role of LayerNorm in Transformers' Attention Shaked Brody Shiyu Jin Xinghao Zhu MoE 59 30 0 04 May 2023
Sparks of Artificial General Intelligence: Early experiments with GPT-4 Sébastien Bubeck Varun Chandrasekaran Ronen Eldan J. Gehrke Eric Horvitz ... Scott M. Lundberg Harsha Nori Hamid Palangi Marco Tulio Ribeiro Yi Zhang ELM AI4MH AI4CE ALM 206 2,232 0 22 Mar 2023
Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations Atticus Geiger Zhengxuan Wu Christopher Potts Thomas F. Icard Noah D. Goodman CML 73 98 0 05 Mar 2023
Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small Kevin Wang Alexandre Variengien Arthur Conmy Buck Shlegeris Jacob Steinhardt 210 486 0 01 Nov 2022
Toy Models of Superposition Nelson Elhage Tristan Hume Catherine Olsson Nicholas Schiefer T. Henighan ... Sam McCandlish Jared Kaplan Dario Amodei Martin Wattenberg C. Olah AAML MILM 120 314 0 21 Sep 2022
Incorporating Residual and Normalization Layers into Analysis of Masked Language Models Goro Kobayashi Tatsuki Kuribayashi Sho Yokoi Kentaro Inui 158 45 0 15 Sep 2021
Probing Classifiers: Promises, Shortcomings, and Advances Yonatan Belinkov 221 402 0 24 Feb 2021
Efficient Estimation of Word Representations in Vector Space Tomáš Mikolov Kai Chen G. Corrado J. Dean 3DV 228 31,150 0 16 Jan 2013