Overcoming a Theoretical Limitation of Self-Attention

24 February 2022

Papers citing "Overcoming a Theoretical Limitation of Self-Attention"

18 / 68 papers shown

Title
Practical Computational Power of Linear Transformers and Their Recurrent and Self-Referential Extensions Kazuki Irie Róbert Csordás Jürgen Schmidhuber 28 11 0 24 Oct 2023
Masked Hard-Attention Transformers Recognize Exactly the Star-Free Languages Andy Yang David Chiang Dana Angluin 28 14 0 21 Oct 2023
From Interpolation to Extrapolation: Complete Length Generalization for Arithmetic Transformers Shaoxiong Duan Yining Shi Wei Xu 20 8 0 18 Oct 2023
Scaling Laws of RoPE-based Extrapolation Xiaoran Liu Hang Yan Shuo Zhang Chen An Xipeng Qiu Dahua Lin 23 6 0 08 Oct 2023
Logical Languages Accepted by Transformer Encoders with Hard Attention Pablo Barceló A. Kozachinskiy A. W. Lin Vladimir Podolskii 25 15 0 05 Oct 2023
Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functions S. Bhattamishra Arkil Patel Phil Blunsom Varun Kanade 19 41 0 04 Oct 2023
Qwen Technical Report Jinze Bai Shuai Bai Yunfei Chu Zeyu Cui Kai Dang ... Zhenru Zhang Chang Zhou Jingren Zhou Xiaohuan Zhou Tianhang Zhu OSLM 29 1,568 0 28 Sep 2023
Evaluating Transformer's Ability to Learn Mildly Context-Sensitive Languages Shunjie Wang Shane Steinert-Threlkeld 25 4 0 02 Sep 2023
Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators? T. Kajitsuka Issei Sato 29 16 0 26 Jul 2023
Exposing Attention Glitches with Flip-Flop Language Modeling Bingbin Liu Jordan T. Ash Surbhi Goel A. Krishnamurthy Cyril Zhang LRM 27 46 0 01 Jun 2023
Randomized Positional Encodings Boost Length Generalization of Transformers Anian Ruoss Grégoire Delétang Tim Genewein Jordi Grau-Moya Róbert Csordás Mehdi Abbana Bennani Shane Legg J. Veness LLMAG 25 99 0 26 May 2023
Transformer Working Memory Enables Regular Language Reasoning and Natural Language Length Extrapolation Ta-Chung Chi Ting-Han Fan Alexander I. Rudnicky Peter J. Ramadge LRM 6 12 0 05 May 2023
Tighter Bounds on the Expressivity of Transformer Encoders David Chiang Peter A. Cholak A. Pillay 27 53 0 25 Jan 2023
Algorithms for Acyclic Weighted Finite-State Automata with Failure Arcs Anej Svete Benjamin Dayan Tim Vieira Ryan Cotterell Jason Eisner 22 1 0 17 Jan 2023
Simplicity Bias in Transformers and their Ability to Learn Sparse Boolean Functions S. Bhattamishra Arkil Patel Varun Kanade Phil Blunsom 14 43 0 22 Nov 2022
Towards A Unified Conformer Structure: from ASR to ASV Task Dexin Liao Tao Jiang Feng Wang Lin Li Q. Hong 22 10 0 14 Nov 2022
State-of-the-art generalisation research in NLP: A taxonomy and review Dieuwke Hupkes Mario Giulianelli Verna Dankers Mikel Artetxe Yanai Elazar ... Leila Khalatbari Maria Ryskina Rita Frieske Ryan Cotterell Zhijing Jin 106 92 0 06 Oct 2022
Neural Networks and the Chomsky Hierarchy Grégoire Delétang Anian Ruoss Jordi Grau-Moya Tim Genewein L. Wenliang ... Chris Cundy Marcus Hutter Shane Legg Joel Veness Pedro A. Ortega UQCV 94 129 0 05 Jul 2022