Data Distributional Properties Drive Emergent In-Context Learning in Transformers

22 April 2022

Pierre Harvey Richemond

J. Mcclelland

Felix Hill

ArXiv PDF HTML

Papers citing "Data Distributional Properties Drive Emergent In-Context Learning in Transformers"

24 / 174 papers shown

Title
Towards Robust Prompts on Vision-Language Models Jindong Gu Ahmad Beirami Xuezhi Wang Alex Beutel Philip H. S. Torr Yao Qin VLM VPVLM 22 8 0 17 Apr 2023
Meta-Learned Models of Cognition Marcel Binz Ishita Dasgupta A. Jagadish M. Botvinick Jane X. Wang Eric Schulz 24 23 0 12 Apr 2023
Why think step by step? Reasoning emerges from the locality of experience Ben Prystawski Michael Y. Li Noah D. Goodman LRM ReLM 16 89 0 07 Apr 2023
The Quantization Model of Neural Scaling Eric J. Michaud Ziming Liu Uzay Girit Max Tegmark MILM 14 77 0 23 Mar 2023
A Theory of Emergent In-Context Learning as Implicit Structure Induction Michael Hahn Navin Goyal LRM 8 73 0 14 Mar 2023
The Learnability of In-Context Learning Noam Wies Yoav Levine Amnon Shashua 110 89 0 14 Mar 2023
The Life Cycle of Knowledge in Big Language Models: A Survey Boxi Cao Hongyu Lin Xianpei Han Le Sun KELM 26 27 0 14 Mar 2023
Numeracy from Literacy: Data Science as an Emergent Skill from Large Language Models David A. Noever Forrest McKee ELM 29 10 0 31 Jan 2023
A Survey on In-context Learning Qingxiu Dong Lei Li Damai Dai Ce Zheng Jingyuan Ma ... Zhiyong Wu Baobao Chang Xu Sun Lei Li Zhifang Sui ReLM AIMat 20 443 0 31 Dec 2022
Contrastive Distillation Is a Sample-Efficient Self-Supervised Loss Policy for Transfer Learning Christopher T. Lengerich Gabriel Synnaeve Amy Zhang Hugh Leather Kurt Shuster Franccois Charton Charysse Redwood SSL OffRL 14 1 0 21 Dec 2022
Emergent Analogical Reasoning in Large Language Models Taylor W. Webb K. Holyoak Hongjing Lu ReLM ELM LRM AI4CE 14 205 0 19 Dec 2022
Transformers learn in-context by gradient descent J. Oswald Eyvind Niklasson E. Randazzo João Sacramento A. Mordvintsev A. Zhmoginov Max Vladymyrov MLT 20 423 0 15 Dec 2022
General-Purpose In-Context Learning by Meta-Learning Transformers Louis Kirsch James Harrison Jascha Narain Sohl-Dickstein Luke Metz 16 72 0 08 Dec 2022
Talking About Large Language Models Murray Shanahan AI4CE 13 236 0 07 Dec 2022
General Intelligence Requires Rethinking Exploration Minqi Jiang Tim Rocktaschel Edward Grefenstette LRM 20 17 0 15 Nov 2022
Two-stage LLM Fine-tuning with Less Specialization and More Generalization Yihan Wang Si Si Daliang Li Michal Lukasik Felix X. Yu Cho-Jui Hsieh Inderjit S Dhillon Sanjiv Kumar 34 29 0 01 Nov 2022
Don't Prompt, Search! Mining-based Zero-Shot Learning with Language Models Mozes van de Kar Mengzhou Xia Danqi Chen Mikel Artetxe 25 19 0 26 Oct 2022
In-context Reinforcement Learning with Algorithm Distillation Michael Laskin Luyu Wang Junhyuk Oh Emilio Parisotto Stephen Spencer ... Ethan A. Brooks Maxime Gazeau Himanshu Sahni Satinder Singh Volodymyr Mnih OffRL 22 120 0 25 Oct 2022
Transformers generalize differently from information stored in context vs in weights Stephanie C. Y. Chan Ishita Dasgupta Junkyung Kim D. Kumaran Andrew Kyle Lampinen Felix Hill 98 45 0 11 Oct 2022
Large Language Models can Implement Policy Iteration Ethan A. Brooks Logan Walls Richard L. Lewis Satinder Singh LM&Ro OffRL 124 21 0 07 Oct 2022
Transformer Neural Processes: Uncertainty-Aware Meta Learning Via Sequence Modeling Tung Nguyen Aditya Grover BDL UQCV 10 99 0 09 Jul 2022
Can Wikipedia Help Offline Reinforcement Learning? Machel Reid Yutaro Yamada S. Gu 3DV RALM OffRL 127 95 0 28 Jan 2022
Systematic human learning and generalization from a brief tutorial with explanatory feedback A. Nam James L. McClelland 11 0 0 10 Jul 2021
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks Chelsea Finn Pieter Abbeel Sergey Levine OOD 237 11,568 0 09 Mar 2017