Grokking of Hierarchical Structure in Vanilla Transformers

Grokking of Hierarchical Structure in Vanilla Transformers

30 May 2023

Pratyusha Sharma

Christopher D. Manning

Papers citing "Grokking of Hierarchical Structure in Vanilla Transformers"

11 / 11 papers shown

Title
Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representations Yize Zhao Tina Behnia V. Vakilian Christos Thrampoulidis 55 8 0 20 Feb 2025
Sneaking Syntax into Transformer Language Models with Tree Regularization Ananjan Nandi Christopher D. Manning Shikhar Murty 74 0 0 28 Nov 2024
On Memorization of Large Language Models in Logical Reasoning Chulin Xie Yangsibo Huang Chiyuan Zhang Da Yu Xinyun Chen Bill Yuchen Lin Bo Li Badih Ghazi Ravi Kumar LRM 51 20 0 30 Oct 2024
Theoretical Insights into Fine-Tuning Attention Mechanism: Generalization and Optimization Xinhao Yao Hongjin Qian Xiaolin Hu Gengze Xu Wei Liu Jian Luan B. Wang Y. Liu 48 0 0 03 Oct 2024
HDT: Hierarchical Document Transformer Haoyu He Markus Flicke Jan Buchmann Iryna Gurevych Andreas Geiger 35 0 0 11 Jul 2024
Beyond Scaling Laws: Understanding Transformer Performance with Associative Memory Xueyan Niu Bo Bai Lei Deng Wei Han 31 6 0 14 May 2024
Learned feature representations are biased by complexity, learning order, position, and more Andrew Kyle Lampinen Stephanie C. Y. Chan Katherine Hermann AI4CE FaML SSL OOD 32 6 0 09 May 2024
Towards a Mechanistic Interpretation of Multi-Step Reasoning Capabilities of Language Models Yifan Hou Jiaoda Li Yu Fei Alessandro Stolfo Wangchunshu Zhou Guangtao Zeng Antoine Bosselut Mrinmaya Sachan LRM 30 39 0 23 Oct 2023
Faith and Fate: Limits of Transformers on Compositionality Nouha Dziri Ximing Lu Melanie Sclar Xiang Lorraine Li Liwei Jian ... Sean Welleck Xiang Ren Allyson Ettinger Zaïd Harchaoui Yejin Choi ReLM LRM 30 328 0 29 May 2023
Omnigrok: Grokking Beyond Algorithmic Data Ziming Liu Eric J. Michaud Max Tegmark 56 76 0 03 Oct 2022
Transformers Generalize Linearly Jackson Petty Robert Frank AI4CE 213 16 0 24 Sep 2021