Pretraining task diversity and the emergence of non-Bayesian in-context
learning for regression

Pretraining task diversity and the emergence of non-Bayesian in-context learning for regression

26 June 2023

Allan Raventós

Papers citing "Pretraining task diversity and the emergence of non-Bayesian in-context learning for regression"

16 / 16 papers shown

Title
On the generalization of language models from in-context learning and finetuning: a controlled study Andrew Kyle Lampinen Arslan Chaudhry Stephanie Chan Cody Wild Diane Wan Alex Ku Jorg Bornschein Razvan Pascanu Murray Shanahan James L. McClelland 46 0 0 01 May 2025
In-Context Learning with Hypothesis-Class Guidance Ziqian Lin Shubham Kumar Bharti Kangwook Lee 69 0 0 27 Feb 2025
In-context learning and Occam's razor Eric Elmoznino Tom Marty Tejas Kasetty Léo Gagnon Sarthak Mittal Mahan Fathi Dhanya Sridhar Guillaume Lajoie 32 1 0 17 Oct 2024
Context-Scaling versus Task-Scaling in In-Context Learning Amirhesam Abedsoltan Adityanarayanan Radhakrishnan Jingfeng Wu M. Belkin ReLM LRM 32 3 0 16 Oct 2024
Bypassing the Exponential Dependency: Looped Transformers Efficiently Learn In-context by Multi-step Gradient Descent Bo Chen Xiaoyu Li Yingyu Liang Zhenmei Shi Zhao-quan Song 86 19 0 15 Oct 2024
In-context Learning in Presence of Spurious Correlations Hrayr Harutyunyan R. Darbinyan Samvel Karapetyan Hrant Khachatrian LRM 40 1 0 04 Oct 2024
Spin glass model of in-context learning Yuhao Li Ruoran Bai Haiping Huang LRM 42 0 0 05 Aug 2024
Towards Better Understanding of In-Context Learning Ability from In-Context Uncertainty Quantification Shang Liu Zhongze Cai Guanting Chen Xiaocheng Li UQCV 38 1 0 24 May 2024
Asymptotic theory of in-context learning by linear attention Yue M. Lu Mary I. Letey Jacob A. Zavatone-Veth Anindita Maiti C. Pehlevan 19 10 0 20 May 2024
Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretraining Licong Lin Yu Bai Song Mei OffRL 30 42 0 12 Oct 2023
A Benchmark for Learning to Translate a New Language from One Grammar Book Garrett Tanzer Mirac Suzgun Chenguang Xi Dan Jurafsky Luke Melas-Kyriazi 24 51 0 28 Sep 2023
In-context Learning and Induction Heads Catherine Olsson Nelson Elhage Neel Nanda Nicholas Joseph Nova Dassarma ... Tom B. Brown Jack Clark Jared Kaplan Sam McCandlish C. Olah 242 458 0 24 Sep 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 306 11,909 0 04 Mar 2022
Multitask Prompted Training Enables Zero-Shot Task Generalization Victor Sanh Albert Webson Colin Raffel Stephen H. Bach Lintang Sutawika ... T. Bers Stella Biderman Leo Gao Thomas Wolf Alexander M. Rush LRM 211 1,656 0 15 Oct 2021
What Makes Good In-Context Examples for GPT- $3$ ? Jiachang Liu Dinghan Shen Yizhe Zhang Bill Dolan Lawrence Carin Weizhu Chen AAML RALM 275 1,312 0 17 Jan 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 248 1,986 0 31 Dec 2020