Explaining grokking through circuit efficiency

5 September 2023

Papers citing "Explaining grokking through circuit efficiency"

44 / 44 papers shown

Title
Quiet Feature Learning in Algorithmic Tasks Prudhviraj Naidu Zixian Wang Leon Bergen R. Paturi VLM 52 0 0 06 May 2025
A Mathematical Philosophy of Explanations in Mechanistic Interpretability -- The Strange Science Part I.i Kola Ayonrinde Louis Jaburi MILM 82 1 0 01 May 2025
A Two-Phase Perspective on Deep Learning Dynamics Robert de Mello Koch Animik Ghosh 34 0 0 17 Apr 2025
Let Me Grok for You: Accelerating Grokking via Embedding Transfer from a Weaker Model Zhiwei Xu Zhiyu Ni Yixin Wang Wei Hu CLL 32 0 0 17 Apr 2025
Low Rank and Sparse Fourier Structure in Recurrent Networks Trained on Modular Addition Akshay Rangamani 40 0 0 28 Mar 2025
All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuning Gokul Swamy Sanjiban Choudhury Wen Sun Zhiwei Steven Wu J. Andrew Bagnell OffRL 42 7 0 03 Mar 2025
Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs Jan Betley Daniel Tan Niels Warncke Anna Sztyber-Betley Xuchan Bao Martín Soto Nathan Labenz Owain Evans AAML 76 8 0 24 Feb 2025
Grokking at the Edge of Numerical Stability Lucas Prieto Melih Barsbey Pedro A.M. Mediano Tolga Birdal 32 3 0 08 Jan 2025
Do Mice Grok? Glimpses of Hidden Progress During Overtraining in Sensory Cortex Tanishq Kumar Blake Bordelon C. Pehlevan Venkatesh N. Murthy Samuel Gershman OOD CLL SSL 48 0 0 05 Nov 2024
Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics Yaniv Nikankin Anja Reusch Aaron Mueller Yonatan Belinkov AIFin LRM 33 21 0 28 Oct 2024
The Fair Language Model Paradox Andrea Pinto Tomer Galanti Randall Balestriero 18 0 0 15 Oct 2024
The Same But Different: Structural Similarities and Differences in Multilingual Language Modeling Ruochen Zhang Qinan Yu Matianyu Zang Carsten Eickhoff Ellie Pavlick 43 1 0 11 Oct 2024
Composing Global Optimizers to Reasoning Tasks via Algebraic Objects in Neural Nets Yuandong Tian 47 0 0 02 Oct 2024
Emergence in non-neural models: grokking modular arithmetic via average gradient outer product Neil Rohit Mallinar Daniel Beaglehole Libin Zhu Adityanarayanan Radhakrishnan Parthe Pandit Misha Belkin 37 7 0 29 Jul 2024
Why Do You Grok? A Theoretical Analysis of Grokking Modular Addition Mohamad Amin Mohamadi Zhiyuan Li Lei Wu Danica J. Sutherland 38 10 0 17 Jul 2024
A Practical Review of Mechanistic Interpretability for Transformer-Based Language Models Daking Rai Yilun Zhou Shi Feng Abulhair Saparov Ziyu Yao 75 18 0 02 Jul 2024
A rationale from frequency perspective for grokking in training neural network Zhangchen Zhou Yaoyu Zhang Z. Xu 38 2 0 24 May 2024
Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization Boshi Wang Xiang Yue Yu-Chuan Su Huan Sun LRM 29 41 0 23 May 2024
Progress Measures for Grokking on Real-world Tasks Satvik Golechha 24 1 0 21 May 2024
Learning Syntax Without Planting Trees: Understanding Hierarchical Generalization in Transformers Kabir Ahuja Vidhisha Balachandran Madhur Panwar Tianxing He Noah A. Smith Navin Goyal Yulia Tsvetkov 27 8 0 25 Apr 2024
Mechanistic Interpretability for AI Safety -- A Review Leonard Bereska E. Gavves AI4CE 38 111 0 22 Apr 2024
σ-GPTs: A New Approach to Autoregressive Models Arnaud Pannatier Evann Courdier Franccois Fleuret AI4TS 26 7 0 15 Apr 2024
Deep Networks Always Grok and Here is Why Ahmed Imtiaz Humayun Randall Balestriero Richard Baraniuk AAML OOD AI4CE 43 19 0 23 Feb 2024
Unified View of Grokking, Double Descent and Emergent Abilities: A Perspective from Circuits Competition Yufei Huang Shengding Hu Xu Han Zhiyuan Liu Maosong Sun 62 14 0 23 Feb 2024
On Catastrophic Inheritance of Large Foundation Models Hao Chen Bhiksha Raj Xing Xie Jindong Wang AI4CE 48 12 0 02 Feb 2024
Critical Data Size of Language Models from a Grokking Perspective Xuekai Zhu Yao Fu Bowen Zhou Zhouhan Lin 17 14 0 19 Jan 2024
Grokking Group Multiplication with Cosets Dashiell Stander Qinan Yu Honglu Fan Stella Biderman 33 9 0 11 Dec 2023
Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokking Kaifeng Lyu Jikai Jin Zhiyuan Li Simon S. Du Jason D. Lee Wei Hu AI4CE 33 32 0 30 Nov 2023
The Transient Nature of Emergent In-Context Learning in Transformers Aaditya K. Singh Stephanie C. Y. Chan Ted Moskovitz Erin Grant Andrew M. Saxe Felix Hill 62 31 0 14 Nov 2023
Understanding Grokking Through A Robustness Viewpoint Zhiquan Tan Weiran Huang AAML OOD 30 6 0 11 Nov 2023
Training Dynamics of Contextual N-Grams in Language Models Lucia Quirke Lovis Heindrich Wes Gurnee Neel Nanda 10 4 0 01 Nov 2023
Bridging Lottery Ticket and Grokking: Understanding Grokking from Inner Structure of Networks Gouki Minegishi Yusuke Iwasawa Yutaka Matsuo 11 3 0 30 Oct 2023
Grokking as the Transition from Lazy to Rich Training Dynamics Tanishq Kumar Blake Bordelon Samuel Gershman C. Pehlevan 28 31 0 09 Oct 2023
Grokking as Compression: A Nonlinear Complexity Perspective Ziming Liu Ziqian Zhong Max Tegmark 30 9 0 09 Oct 2023
Grokking as a First Order Phase Transition in Two Layer Networks Noa Rubin Inbar Seroussi Z. Ringel 24 15 0 05 Oct 2023
Predicting Emergent Abilities with Infinite Resolution Evaluation Shengding Hu Xin Liu Xu Han Xinrong Zhang Chaoqun He ... Ning Ding Zebin Ou Guoyang Zeng Zhiyuan Liu Maosong Sun ELM LRM 20 13 0 05 Oct 2023
Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Data Zhiwei Xu Yutong Wang Spencer Frei Gal Vardi Wei Hu MLT 24 23 0 04 Oct 2023
Towards Best Practices of Activation Patching in Language Models: Metrics and Methods Fred Zhang Neel Nanda LLMSV 26 96 0 27 Sep 2023
Faith and Fate: Limits of Transformers on Compositionality Nouha Dziri Ximing Lu Melanie Sclar Xiang Lorraine Li Liwei Jian ... Sean Welleck Xiang Ren Allyson Ettinger Zaïd Harchaoui Yejin Choi ReLM LRM 28 327 0 29 May 2023
Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small Kevin Wang Alexandre Variengien Arthur Conmy Buck Shlegeris Jacob Steinhardt 210 491 0 01 Nov 2022
Omnigrok: Grokking Beyond Algorithmic Data Ziming Liu Eric J. Michaud Max Tegmark 54 76 0 03 Oct 2022
In-context Learning and Induction Heads Catherine Olsson Nelson Elhage Neel Nanda Nicholas Joseph Nova Dassarma ... Tom B. Brown Jack Clark Jared Kaplan Sam McCandlish C. Olah 240 456 0 24 Sep 2022
Toy Models of Superposition Nelson Elhage Tristan Hume Catherine Olsson Nicholas Schiefer T. Henighan ... Sam McCandlish Jared Kaplan Dario Amodei Martin Wattenberg C. Olah AAML MILM 120 316 0 21 Sep 2022
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 226 4,424 0 23 Jan 2020