Randomized Positional Encodings Boost Length Generalization of
Transformers

Randomized Positional Encodings Boost Length Generalization of Transformers

26 May 2023

Grégoire Delétang

Jordi Grau-Moya

Róbert Csordás

Mehdi Abbana Bennani

Papers citing "Randomized Positional Encodings Boost Length Generalization of Transformers"

15 / 15 papers shown

Title
Spline-based Transformers Prashanth Chandran Agon Serifi Markus Gross Moritz Bächer 36 0 0 03 Apr 2025
TRA: Better Length Generalisation with Threshold Relative Attention Mattia Opper Roland Fernandez P. Smolensky Jianfeng Gao 39 0 0 29 Mar 2025
Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challenges Nayoung Lee Ziyang Cai Avi Schwarzschild Kangwook Lee Dimitris Papailiopoulos ReLM VLM LRM AI4CE 73 4 0 03 Feb 2025
Investigating Length Issues in Document-level Machine Translation Ziqian Peng Rachel Bawden François Yvon 69 1 0 23 Dec 2024
Training Neural Networks as Recognizers of Formal Languages Alexandra Butoi Ghazal Khalighinejad Anej Svete Josef Valvoda Ryan Cotterell Brian DuSell NAI 36 2 0 11 Nov 2024
LayerDAG: A Layerwise Autoregressive Diffusion Model for Directed Acyclic Graph Generation Mufei Li Viraj Shitole Eli Chien Changhai Man Zhaodong Wang Srinivas Sridharan Ying Zhang Tushar Krishna P. Li 30 0 0 04 Nov 2024
TULIP: Token-length Upgraded CLIP Ivona Najdenkoska Mohammad Mahdi Derakhshani Yuki M. Asano N. V. Noord Marcel Worring Cees G. M. Snoek VLM 46 3 0 13 Oct 2024
Round and Round We Go! What makes Rotary Positional Encodings useful? Federico Barbero Alex Vitvitskyi Christos Perivolaropoulos Razvan Pascanu Petar Velickovic 75 16 0 08 Oct 2024
The CLRS-Text Algorithmic Reasoning Language Benchmark Larisa Markeeva Sean McLeish Borja Ibarz Wilfried Bounsi Olga Kozlova Alex Vitvitskyi Charles Blundell Tom Goldstein Avi Schwarzschild Petar Veličković LRM 34 12 0 06 Jun 2024
VSTAR: Generative Temporal Nursing for Longer Dynamic Video Synthesis Yumeng Li William H. Beluch M. Keuper Dan Zhang Anna Khoreva DiffM VGen 74 5 0 20 Mar 2024
Language Modeling Is Compression Grégoire Delétang Anian Ruoss Paul-Ambroise Duquenne Elliot Catt Tim Genewein ... Wenliang Kevin Li Matthew Aitchison Laurent Orseau Marcus Hutter J. Veness AI4CE 30 129 0 19 Sep 2023
Learning a Fourier Transform for Linear Relative Positional Encodings in Transformers K. Choromanski Shanda Li Valerii Likhosherstov Kumar Avinava Dubey Shengjie Luo Di He Yiming Yang Tamás Sarlós Thomas Weingarten Adrian Weller 19 8 0 03 Feb 2023
Systematic Generalization and Emergent Structures in Transformers Trained on Structured Tasks Yuxuan Li James L. McClelland 29 17 0 02 Oct 2022
Neural Networks and the Chomsky Hierarchy Grégoire Delétang Anian Ruoss Jordi Grau-Moya Tim Genewein L. Wenliang ... Chris Cundy Marcus Hutter Shane Legg Joel Veness Pedro A. Ortega UQCV 94 129 0 05 Jul 2022
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press Noah A. Smith M. Lewis 245 695 0 27 Aug 2021