The Evolution of Statistical Induction Heads: In-Context Learning Markov
Chains

The Evolution of Statistical Induction Heads: In-Context Learning Markov Chains

16 February 2024

Benjamin L. Edelman

Nikolaos Tsilivis

Papers citing "The Evolution of Statistical Induction Heads: In-Context Learning Markov Chains"

13 / 13 papers shown

Title
Quiet Feature Learning in Algorithmic Tasks Prudhviraj Naidu Zixian Wang Leon Bergen R. Paturi VLM 52 0 0 06 May 2025
HyperTree Planning: Enhancing LLM Reasoning via Hierarchical Thinking Runquan Gui Z. Wang J. Wang Chi Ma Huiling Zhen M. Yuan Jianye Hao Defu Lian Enhong Chen Feng Wu LRM 51 0 0 05 May 2025
Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representations Yize Zhao Tina Behnia V. Vakilian Christos Thrampoulidis 55 7 0 20 Feb 2025
From Markov to Laplace: How Mamba In-Context Learns Markov Chains Marco Bondaschi Nived Rajaraman Xiuying Wei Kannan Ramchandran Razvan Pascanu Çağlar Gülçehre Michael C. Gastpar Ashok Vardhan Makkuva 58 0 0 17 Feb 2025
Are Transformers Able to Reason by Connecting Separated Knowledge in Training Data? Yutong Yin Zhaoran Wang LRM ReLM 50 0 0 27 Jan 2025
N-Gram Induction Heads for In-Context RL: Improving Stability and Reducing Data Needs Ilya Zisman Alexander Nikulin Andrei Polubarov Nikita Lyubaykin Vladislav Kurenkov Andrei Polubarov Igor Kiselev Vladislav Kurenkov OffRL 44 1 0 04 Nov 2024
Toward Understanding In-context vs. In-weight Learning Bryan Chan Xinyi Chen András Gyorgy Dale Schuurmans 65 3 0 30 Oct 2024
Transformers Handle Endogeneity in In-Context Linear Regression Haodong Liang Krishnakumar Balasubramanian Lifeng Lai 32 1 0 02 Oct 2024
Representing Rule-based Chatbots with Transformers Dan Friedman Abhishek Panigrahi Danqi Chen 56 1 0 15 Jul 2024
A Practical Review of Mechanistic Interpretability for Transformer-Based Language Models Daking Rai Yilun Zhou Shi Feng Abulhair Saparov Ziyu Yao 70 18 0 02 Jul 2024
How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression? Jingfeng Wu Difan Zou Zixiang Chen Vladimir Braverman Quanquan Gu Peter L. Bartlett 116 48 0 12 Oct 2023
SGD learning on neural networks: leap complexity and saddle-to-saddle dynamics Emmanuel Abbe Enric Boix-Adserà Theodor Misiakiewicz FedML MLT 76 72 0 21 Feb 2023
In-context Learning and Induction Heads Catherine Olsson Nelson Elhage Neel Nanda Nicholas Joseph Nova Dassarma ... Tom B. Brown Jack Clark Jared Kaplan Sam McCandlish C. Olah 240 453 0 24 Sep 2022