Do pretrained Transformers Learn In-Context by Gradient Descent?

v1v2v3v4v5 (latest)

Do pretrained Transformers Learn In-Context by Gradient Descent?

12 October 2023

Lingfeng Shen

Daniel Khashabi

ArXiv (abs)PDF HTML

Papers citing "Do pretrained Transformers Learn In-Context by Gradient Descent?"

10 / 10 papers shown

Title
Genomic Next-Token Predictors are In-Context Learners Nathan Breslow Aayush Mishra Mahler Revsine Michael C. Schatz Anqi Liu Daniel Khashabi 191 0 0 16 Nov 2025
IA2: Alignment with ICL Activations Improves Supervised Fine-Tuning Aayush Mishra Daniel Khashabi Anqi Liu 140 1 0 26 Sep 2025
Relational reasoning and inductive bias in transformers trained on a transitive inference task J. Geerts Stephanie Chan Claudia Clopath Kimberly L. Stachenfeld LRM 154 2 0 04 Jun 2025
ICL CIPHERS: Quantifying "Learning" in In-Context Learning via Substitution Ciphers Zhouxiang Fang Aayush Mishra Muhan Gao Anqi Liu Daniel Khashabi 361 2 0 28 Apr 2025
Scaling sparse feature circuit finding for in-context learning Dmitrii Kharlapenko Shivalika Singh Fazl Barez Arthur Conmy Neel Nanda 256 3 0 18 Apr 2025
Deeper Insights Without Updates: The Power of In-Context Learning Over Fine-TuningConference on Empirical Methods in Natural Language Processing (EMNLP), 2024 Qingyu Yin Xuzheng He Luoao Deng Chak Tou Leong Fan Wang Yanzhao Yan Xiaoyu Shen Qiang Zhang 322 9 0 07 Oct 2024
What Do Language Models Learn in Context? The Structured Task HypothesisAnnual Meeting of the Association for Computational Linguistics (ACL), 2024 Jiaoda Li Buse Giledereli Mrinmaya Sachan Robert Bamler LRM 275 14 0 06 Jun 2024
The Mystery of In-Context Learning: A Comprehensive Survey on Interpretation and AnalysisConference on Empirical Methods in Natural Language Processing (EMNLP), 2023 Yuxiang Zhou Jiazheng Li Yanzheng Xiang Hanqi Yan Lin Gui Yulan He 283 31 0 01 Nov 2023
The Expressibility of Polynomial based Attention Scheme Zhao Song Guangyi Xu Junze Yin 278 7 0 30 Oct 2023
A Survey on In-context LearningConference on Empirical Methods in Natural Language Processing (EMNLP), 2022 Qingxiu Dong Lei Li Damai Dai Ce Zheng Jingyuan Ma ... Zhiyong Wu Baobao Chang Xu Sun Lei Li Zhifang Sui ReLM AIMat 424 834 0 31 Dec 2022