Eureka-Moments in Transformers: Multi-Step Tasks Reveal Softmax Induced
Optimization Problems

Eureka-Moments in Transformers: Multi-Step Tasks Reveal Softmax Induced Optimization Problems

19 October 2023

David T. Hoffmann

Jelena Bratulić

Nadine Behrmann

Thomas Brox

Papers citing "Eureka-Moments in Transformers: Multi-Step Tasks Reveal Softmax Induced Optimization Problems"

5 / 5 papers shown

Title
Stabilizing Transformer Training by Preventing Attention Entropy Collapse Shuangfei Zhai Tatiana Likhomanenko Etai Littwin Dan Busbridge Jason Ramapuram Yizhe Zhang Jiatao Gu J. Susskind AAML 38 64 0 11 Mar 2023
Accumulated Trivial Attention Matters in Vision Transformers on Small Datasets Xiangyu Chen Qinghao Hu Kaidong Li Cuncong Zhong Guanghui Wang ViT 33 11 0 22 Oct 2022
Omnigrok: Grokking Beyond Algorithmic Data Ziming Liu Eric J. Michaud Max Tegmark 54 76 0 03 Oct 2022
In-context Learning and Induction Heads Catherine Olsson Nelson Elhage Neel Nanda Nicholas Joseph Nova Dassarma ... Tom B. Brown Jack Clark Jared Kaplan Sam McCandlish C. Olah 240 456 0 24 Sep 2022
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 243 1,815 0 17 Sep 2019