Data Distributional Properties Drive Emergent In-Context Learning in Transformers

22 April 2022

Pierre Harvey Richemond

J. Mcclelland

Felix Hill

ArXiv PDF HTML

Papers citing "Data Distributional Properties Drive Emergent In-Context Learning in Transformers"

50 / 174 papers shown

Title
Comet: Accelerating Private Inference for Large Language Model by Predicting Activation Sparsity Guang Yan Yuhui Zhang Zimu Guo Lutan Zhao Xiaojun Chen Chen Wang Wenhao Wang Dan Meng Rui Hou 19 0 0 12 May 2025
On the generalization of language models from in-context learning and finetuning: a controlled study Andrew Kyle Lampinen Arslan Chaudhry Stephanie Chan Cody Wild Diane Wan Alex Ku Jorg Bornschein Razvan Pascanu Murray Shanahan James L. McClelland 46 0 0 01 May 2025
In-Context Learning can distort the relationship between sequence likelihoods and biological fitness Pranav Kantroo Günter P. Wagner Benjamin B. Machta 34 0 0 23 Apr 2025
Implicit In-Context Learning: Evidence from Artificial Language Experiments Xiaomeng Ma Qihui Xu ReLM 48 0 0 31 Mar 2025
Factored Agents: Decoupling In-Context Learning and Memorization for Robust Tool Use Nicholas Roth Christopher Hidey Lucas Spangher William Arnold Chang Ye Nick Masiewicki Jinoo Baek Peter Grabowski Eugene Ie LLMAG 48 0 0 29 Mar 2025
How do language models learn facts? Dynamics, curricula and hallucinations Nicolas Zucchet J. Bornschein Stephanie C. Y. Chan Andrew Kyle Lampinen Razvan Pascanu Soham De KELM HILM LRM 69 1 1 27 Mar 2025
Transformer-based Wireless Symbol Detection Over Fading Channels Li Fan Jing Yang Cong Shen 34 0 0 20 Mar 2025
MergeQuant: Accurate 4-bit Static Quantization of Large Language Models by Channel-wise Calibration Jinguang Wang J. Wang Haifeng Sun Tingting Yang Zirui Zhuang Wanyi Ning Yuexi Yin Q. Qi Jianxin Liao MQ MoMe 44 0 0 07 Mar 2025
Strategy Coopetition Explains the Emergence and Transience of In-Context Learning Aaditya K. Singh Ted Moskovitz Sara Dragutinovic Felix Hill Stephanie C. Y. Chan Andrew Saxe 58 0 0 07 Mar 2025
Data Distributional Properties As Inductive Bias for Systematic Generalization Felipe del-Rio Alain Raymond-Sáez Daniel Florea Rodrigo Toro Icarte Julio Hurtado Cristian B. Calderon Á. Soto AI4CE 28 0 0 27 Feb 2025
General Reasoning Requires Learning to Reason from the Get-go Seungwook Han Jyothish Pari Samuel J. Gershman Pulkit Agrawal LRM 63 0 0 26 Feb 2025
Towards Auto-Regressive Next-Token Prediction: In-Context Learning Emerges from Generalization Zixuan Gong Xiaolin Hu Huayi Tang Yong Liu 33 0 0 24 Feb 2025
In-context learning of evolving data streams with tabular foundational models Afonso Lourenço João Gama Eric P. Xing Goreti Marreiros 53 0 0 24 Feb 2025
Nonlinear dynamics of localization in neural receptive fields Leon Lufkin Andrew M. Saxe Erin Grant 40 2 0 28 Jan 2025
Training Dynamics of In-Context Learning in Linear Attention Yedi Zhang Aaditya K. Singh Peter E. Latham Andrew Saxe MLT 53 1 0 28 Jan 2025
What Matters for In-Context Learning: A Balancing Act of Look-up and In-Weight Learning Jelena Bratulić Sudhanshu Mittal Christian Rupprecht Thomas Brox 29 0 0 09 Jan 2025
Wasserstein Spatial Depth F. Bachoc Alberto González Sanz Jean-Michel Loubes Yisha Yao 24 1 0 16 Nov 2024
RuAG: Learned-rule-augmented Generation for Large Language Models Yudi Zhang Pei Xiao Lu Wang C. Zhang Meng Fang ... Qingwei Lin Mykola Pechenizkiy Dongmei Zhang Saravan Rajmohan Qi Zhang LRM 38 0 0 04 Nov 2024
Toward Understanding In-context vs. In-weight Learning Bryan Chan Xinyi Chen András Gyorgy Dale Schuurmans 62 3 0 30 Oct 2024
On the Role of Depth and Looping for In-Context Learning with Task Diversity Khashayar Gatmiry Nikunj Saunshi Sashank J. Reddi Stefanie Jegelka Sanjiv Kumar 18 2 0 29 Oct 2024
In-context learning and Occam's razor Eric Elmoznino Tom Marty Tejas Kasetty Léo Gagnon Sarthak Mittal Mahan Fathi Dhanya Sridhar Guillaume Lajoie 29 1 0 17 Oct 2024
Aggregation Artifacts in Subjective Tasks Collapse Large Language Models' Posteriors Georgios Chochlakis Alexandros Potamianos Kristina Lerman Shrikanth Narayanan 25 0 0 17 Oct 2024
Neural networks that overcome classic challenges through practice Kazuki Irie Brenden M. Lake 29 4 0 14 Oct 2024
ELICIT: LLM Augmentation via External In-Context Capability Futing Wang Jianhao Yan Yue Zhang Tao Lin 35 0 0 12 Oct 2024
On-Chip Learning via Transformer In-Context Learning Jan Finkbeiner Emre Neftci 16 0 0 11 Oct 2024
Retrieval-Augmented Decision Transformer: External Memory for In-context RL Thomas Schmied Fabian Paischer Vihang Patil M. Hofmarcher Razvan Pascanu Sepp Hochreiter OffRL 26 6 0 09 Oct 2024
Everything Everywhere All at Once: LLMs can In-Context Learn Multiple Tasks in Superposition Zheyang Xiong Ziyang Cai John Cooper Albert Ge Vasilis Papageorgiou ... Saurabh Agarwal Grigorios G Chrysos Samet Oymak Kangwook Lee Dimitris Papailiopoulos LRM 22 1 0 08 Oct 2024
From Sparse Dependence to Sparse Attention: Unveiling How Chain-of-Thought Enhances Transformer Sample Efficiency Kaiyue Wen Huaqing Zhang Hongzhou Lin Jingzhao Zhang MoE LRM 52 2 0 07 Oct 2024
Task Diversity Shortens the ICL Plateau Jaeyeon Kim Sehyun Kwon Joo Young Choi Jongho Park Jaewoong Cho Jason D. Lee Ernest K. Ryu MoMe 29 2 0 07 Oct 2024
GAMformer: In-Context Learning for Generalized Additive Models Andreas Mueller Julien N. Siems Harsha Nori David Salinas Arber Zela Rich Caruana Frank Hutter AI4CE 31 3 0 06 Oct 2024
Wrong-of-Thought: An Integrated Reasoning Framework with Multi-Perspective Verification and Wrong Information Yongheng Zhang Qiguang Chen Jingxuan Zhou Peng Wang Jiasheng Si Jin Wang Wenpeng Lu Libo Qin LRM 41 3 0 06 Oct 2024
In-context Learning in Presence of Spurious Correlations Hrayr Harutyunyan R. Darbinyan Samvel Karapetyan Hrant Khachatrian LRM 30 1 0 04 Oct 2024
Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficient George Wang Jesse Hoogland Stan van Wingerden Zach Furman Daniel Murfet OffRL 15 7 0 03 Oct 2024
Larger Language Models Don't Care How You Think: Why Chain-of-Thought Prompting Fails in Subjective Tasks Georgios Chochlakis Niyantha Maruthu Pandiyan Kristina Lerman Shrikanth Narayanan ReLM KELM LRM 24 0 0 10 Sep 2024
Rule Extrapolation in Language Models: A Study of Compositional Generalization on OOD Prompts Anna Mészáros Szilvia Ujváry Wieland Brendel Patrik Reizinger Ferenc Huszár 24 0 0 09 Sep 2024
DataSculpt: Crafting Data Landscapes for Long-Context LLMs through Multi-Objective Partitioning Keer Lu Xiaonan Nie Zheng Liang Da Pan Shusen Zhang ... Weipeng Chen Zenan Zhou Guosheng Dong Bin Cui Wentao Zhang 27 0 0 02 Sep 2024
Representing Rule-based Chatbots with Transformers Dan Friedman Abhishek Panigrahi Danqi Chen 52 1 0 15 Jul 2024
A Practical Review of Mechanistic Interpretability for Transformer-Based Language Models Daking Rai Yilun Zhou Shi Feng Abulhair Saparov Ziyu Yao 57 18 0 02 Jul 2024
Deciphering the Factors Influencing the Efficacy of Chain-of-Thought: Probability, Memorization, and Noisy Reasoning Akshara Prabhakar Thomas L. Griffiths R. Thomas McCoy LRM 34 16 0 01 Jul 2024
Enhancing In-Context Learning via Implicit Demonstration Augmentation Xiaoling Zhou Wei Ye Yidong Wang Chaoya Jiang Zhemg Lee Rui Xie Shikun Zhang 20 3 0 27 Jun 2024
OutlierTune: Efficient Channel-Wise Quantization for Large Language Models Jinguang Wang Yuexi Yin Haifeng Sun Qi Qi Jingyu Wang Zirui Zhuang Tingting Yang Jianxin Liao 27 2 0 27 Jun 2024
IRCAN: Mitigating Knowledge Conflicts in LLM Generation via Identifying and Reweighting Context-Aware Neurons Dan Shi Renren Jin Tianhao Shen Weilong Dong Xinwei Wu Deyi Xiong 23 2 0 26 Jun 2024
In-Context Learning of Energy Functions Rylan Schaeffer Mikail Khona Sanmi Koyejo 29 0 0 18 Jun 2024
XLand-100B: A Large-Scale Multi-Task Dataset for In-Context Reinforcement Learning Alexander Nikulin Ilya Zisman Alexey Zemtsov Viacheslav Sinii 89 4 0 13 Jun 2024
In-Context Learning of Physical Properties: Few-Shot Adaptation to Out-of-Distribution Molecular Graphs Grzegorz Kaszuba Amirhossein D. Naghdi Dario Massa Stefanos Papanikolaou Andrzej Jaszkiewicz Piotr Sankowski AI4CE OODD 27 0 0 03 Jun 2024
Does learning the right latent variables necessarily improve in-context learning? Sarthak Mittal Eric Elmoznino Léo Gagnon Sangnie Bhardwaj Dhanya Sridhar Guillaume Lajoie 19 0 0 29 May 2024
Language Models Trained to do Arithmetic Predict Human Risky and Intertemporal Choice Jian-Qiao Zhu Haijiang Yan Thomas L. Griffiths 77 2 0 29 May 2024
IM-Context: In-Context Learning for Imbalanced Regression Tasks Ismail Nejjar Faez Ahmed Olga Fink 19 0 0 28 May 2024
On Mesa-Optimization in Autoregressively Trained Transformers: Emergence and Capability Chenyu Zheng Wei Huang Rongzheng Wang Guoqiang Wu Jun Zhu Chongxuan Li 26 1 0 27 May 2024
From Frege to chatGPT: Compositionality in language, cognition, and deep neural networks Jacob Russin Sam Whitman McGrath Danielle J. Williams Lotem Elber-Dorozko AI4CE 59 2 0 24 May 2024