Data Distributional Properties Drive Emergent In-Context Learning in Transformers

22 April 2022

Pierre Harvey Richemond

J. Mcclelland

Felix Hill

ArXiv PDF HTML

Papers citing "Data Distributional Properties Drive Emergent In-Context Learning in Transformers"

50 / 174 papers shown

Title
How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representations Tianyu Guo Wei Hu Song Mei Huan Wang Caiming Xiong Silvio Savarese Yu Bai 11 46 0 16 Oct 2023
Do pretrained Transformers Learn In-Context by Gradient Descent? Lingfeng Shen Aayush Mishra Daniel Khashabi 17 7 0 12 Oct 2023
Lag-Llama: Towards Foundation Models for Probabilistic Time Series Forecasting Kashif Rasul Arjun Ashok Andrew Robert Williams Hena Ghonia Rishika Bhagwatkar ... Nicolas Chapados Alexandre Drouin Valentina Zantedeschi Yuriy Nevmyvaka Irina Rish AI4TS BDL 8 42 0 12 Oct 2023
Is attention required for ICL? Exploring the Relationship Between Model Architecture and In-Context Learning Ability Ivan Lee Nan Jiang Taylor Berg-Kirkpatrick 21 12 0 12 Oct 2023
Towards Better Chain-of-Thought Prompting Strategies: A Survey Zihan Yu Liang He Zhen Wu Xinyu Dai Jiajun Chen LRM 118 40 0 08 Oct 2023
The Cost of Down-Scaling Language Models: Fact Recall Deteriorates before In-Context Learning Tian Jin Nolan Clement Xin Dong Vaishnavh Nagarajan Michael Carbin Jonathan Ragan-Kelley Gintare Karolina Dziugaite LRM 27 5 0 07 Oct 2023
Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functions S. Bhattamishra Arkil Patel Phil Blunsom Varun Kanade 13 40 0 04 Oct 2023
Understanding In-Context Learning from Repetitions Jianhao Yan Jin Xu Chiyu Song Chenming Wu Yafu Li Yue Zhang 17 11 0 30 Sep 2023
HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models Cheng Chen Yuchen Hu Chao-Han Huck Yang Sabato Marco Siniscalchi Pin-Yu Chen E. Chng 8 42 0 27 Sep 2023
Understanding Catastrophic Forgetting in Language Models via Implicit Inference Suhas Kotha Jacob Mitchell Springer Aditi Raghunathan CLL 18 56 0 18 Sep 2023
Cognitive Mirage: A Review of Hallucinations in Large Language Models Hongbin Ye Tong Liu Aijia Zhang Wei Hua Weiqiang Jia HILM 31 76 0 13 Sep 2023
Breaking through the learning plateaus of in-context learning in Transformer Jingwen Fu Tao Yang Yuwang Wang Yan Lu Nanning Zheng 22 0 0 12 Sep 2023
Uncovering mesa-optimization algorithms in Transformers J. Oswald Eyvind Niklasson Maximilian Schlegel Seijin Kobayashi Nicolas Zucchet ... Mark Sandler Blaise Agüera y Arcas Max Vladymyrov Razvan Pascanu João Sacramento 11 53 0 11 Sep 2023
Are Emergent Abilities in Large Language Models just In-Context Learning? Sheng Lu Irina Bigoulaeva Rachneet Sachdeva Harish Tayyar Madabushi Iryna Gurevych LRM ELM ReLM 49 89 0 04 Sep 2023
Gated recurrent neural networks discover attention Nicolas Zucchet Seijin Kobayashi Yassir Akram J. Oswald Maxime Larcher Angelika Steger João Sacramento 23 8 0 04 Sep 2023
Inductive-bias Learning: Generating Code Models with Large Language Model Toma Tanaka Naofumi Emoto Tsukasa Yumibayashi AI4CE 10 0 0 19 Aug 2023
Position: Key Claims in LLM Research Have a Long Tail of Footnotes Anna Rogers A. Luccioni 35 19 0 14 Aug 2023
Large Language Model Displays Emergent Ability to Interpret Novel Literary Metaphors Nicholas Ichien Duvsan Stamenković K. Holyoak LRM 33 4 0 03 Aug 2023
Does Progress On Object Recognition Benchmarks Improve Real-World Generalization? Megan Richards Polina Kirichenko Diane Bouchacourt Mark Ibrahim VLM 64 11 0 24 Jul 2023
In-Context Learning Learns Label Relationships but Is Not Conventional Learning Jannik Kossen Y. Gal Tom Rainforth 25 27 0 23 Jul 2023
SINC: Self-Supervised In-Context Learning for Vision-Language Tasks Yi-Syuan Chen Yun-Zhu Song Cheng Yu Yeo Bei Liu Jianlong Fu Hong-Han Shuai VLM LRM 18 4 0 15 Jul 2023
Large Language Models as General Pattern Machines Suvir Mirchandani F. Xia Peter R. Florence Brian Ichter Danny Driess Montse Gonzalez Arenas Kanishka Rao Dorsa Sadigh Andy Zeng LLMAG 37 183 0 10 Jul 2023
Trainable Transformer in Transformer A. Panigrahi Sadhika Malladi Mengzhou Xia Sanjeev Arora VLM 16 12 0 03 Jul 2023
Understanding In-Context Learning via Supportive Pretraining Data Xiaochuang Han Daniel Simig Todor Mihaylov Yulia Tsvetkov Asli Celikyilmaz Tianlu Wang AIMat 26 33 0 26 Jun 2023
Pretraining task diversity and the emergence of non-Bayesian in-context learning for regression Allan Raventós Mansheej Paul F. Chen Surya Ganguli 19 70 0 26 Jun 2023
Supervised Pretraining Can Learn In-Context Reinforcement Learning Jonathan Lee Annie Xie Aldo Pacchiano Yash Chandak Chelsea Finn Ofir Nachum Emma Brunskill OffRL 17 73 0 26 Jun 2023
Towards Regulatable AI Systems: Technical Gaps and Policy Opportunities Xudong Shen H. Brown Jiashu Tao Martin Strobel Yao Tong Akshay Narayan Harold Soh Finale Doshi-Velez 11 2 0 22 Jun 2023
Schema-learning and rebinding as mechanisms of in-context learning and emergence Siva K. Swaminathan Antoine Dedieu Rajkumar Vasudeva Raju Murray Shanahan Miguel Lazaro-Gredilla Dileep George 18 8 0 16 Jun 2023
Artificial General Intelligence for Medical Imaging Xiang Li Lu Zhang Zihao Wu Zheng Liu Lin Zhao ... Pingkuan Yan Quanzheng Li W. Liu Tianming Liu Dinggang Shen LM&MA AI4CE 17 39 0 08 Jun 2023
In-Context Learning through the Bayesian Prism Madhuri Panwar Kabir Ahuja Navin Goyal BDL 22 38 0 08 Jun 2023
Birth of a Transformer: A Memory Viewpoint A. Bietti Vivien A. Cabannes Diane Bouchacourt Hervé Jégou Léon Bottou 16 80 0 01 Jun 2023
Honey, I Shrunk the Language: Language Model Behavior at Reduced Scale Vijeta Deshpande Dan Pechi Shree Thatte Vladislav Lialin Anna Rumshisky 60 7 0 26 May 2023
Im-Promptu: In-Context Composition from Image Prompts Bhishma Dedhia Michael Chang Jake C. Snell Thomas L. Griffiths N. Jha LRM MLLM 17 1 0 26 May 2023
Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time Zichang Liu Aditya Desai Fangshuo Liao Weitao Wang Victor Xie Zhaozhuo Xu Anastasios Kyrillidis Anshumali Shrivastava 8 200 0 26 May 2023
A Mechanism for Sample-Efficient In-Context Learning for Sparse Retrieval Tasks Jacob D. Abernethy Alekh Agarwal T. V. Marinov Manfred K. Warmuth 8 17 0 26 May 2023
Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective Guhao Feng Bohang Zhang Yuntian Gu Haotian Ye Di He Liwei Wang LRM 21 210 0 24 May 2023
Self-ICL: Zero-Shot In-Context Learning with Self-Generated Demonstrations Wei-Lin Chen Cheng-Kuang Wu Yun-Nung Chen Hsin-Hsi Chen 8 27 0 24 May 2023
Trusting Your Evidence: Hallucinate Less with Context-aware Decoding Weijia Shi Xiaochuang Han M. Lewis Yulia Tsvetkov Luke Zettlemoyer Scott Yih HILM 11 113 0 24 May 2023
Modeling rapid language learning by distilling Bayesian priors into artificial neural networks R. Thomas McCoy Thomas L. Griffiths BDL 31 14 0 24 May 2023
Concept-aware Training Improves In-context Learning Ability of Language Models Michal Štefánik Marek Kadlcík KELM LRM 19 0 0 23 May 2023
Meta-in-context learning in large language models Julian Coda-Forno Marcel Binz Zeynep Akata M. Botvinick Jane X. Wang Eric Schulz LRM 184 34 0 22 May 2023
Explaining Emergent In-Context Learning as Kernel Regression Chi Han Ziqi Wang H. Zhao Heng Ji LRM 19 11 0 22 May 2023
Temporal Knowledge Graph Forecasting Without Knowledge Using In-Context Learning Dong-Ho Lee Kian Ahrabian Woojeong Jin Fred Morstatter Jay Pujara 17 31 0 17 May 2023
What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learning Jane Pan Tianyu Gao Howard Chen Danqi Chen 20 107 0 16 May 2023
Pre-Training to Learn in Context Yuxian Gu Li Dong Furu Wei Minlie Huang CLIP LRM ReLM 106 37 0 16 May 2023
Accelerating Neural Self-Improvement via Bootstrapping Kazuki Irie Jürgen Schmidhuber 19 1 0 02 May 2023
Are Emergent Abilities of Large Language Models a Mirage? Rylan Schaeffer Brando Miranda Oluwasanmi Koyejo LRM 28 262 0 28 Apr 2023
DataComp: In search of the next generation of multimodal datasets S. Gadre Gabriel Ilharco Alex Fang J. Hayase Georgios Smyrnis ... A. Dimakis J. Jitsev Y. Carmon Vaishaal Shankar Ludwig Schmidt VLM 15 404 0 27 Apr 2023
Injecting structural hints: Using language models to study inductive biases in language learning Isabel Papadimitriou Dan Jurafsky 14 12 0 25 Apr 2023
A Latent Space Theory for Emergent Abilities in Large Language Models Hui Jiang LRM 21 35 0 19 Apr 2023