One-layer transformers fail to solve the induction heads task

26 August 2024

Papers citing "One-layer transformers fail to solve the induction heads task"

7 / 7 papers shown

Title
Concise One-Layer Transformers Can Do Function Evaluation (Sometimes) Lena Strobl Dana Angluin Robert Frank 38 0 0 28 Mar 2025
Real-Time Personalization with Simple Transformers Lin An Andrew A. Li Vaisnavi Nemala Gabriel Visotsky 26 0 0 01 Mar 2025
From Markov to Laplace: How Mamba In-Context Learns Markov Chains Marco Bondaschi Nived Rajaraman Xiuying Wei Kannan Ramchandran Razvan Pascanu Çağlar Gülçehre Michael C. Gastpar Ashok Vardhan Makkuva 58 0 0 17 Feb 2025
Theoretical limitations of multi-layer Transformer Lijie Chen Binghui Peng Hongxun Wu AI4CE 67 6 0 04 Dec 2024
Toward Understanding In-context vs. In-weight Learning Bryan Chan Xinyi Chen András Gyorgy Dale Schuurmans 65 3 0 30 Oct 2024
ENTP: Encoder-only Next Token Prediction Ethan Ewer Daewon Chae Thomas Zeng Jinkyu Kim Kangwook Lee 25 3 0 02 Oct 2024
In-context Learning and Induction Heads Catherine Olsson Nelson Elhage Neel Nanda Nicholas Joseph Nova Dassarma ... Tom B. Brown Jack Clark Jared Kaplan Sam McCandlish C. Olah 240 456 0 24 Sep 2022