Length Generalization of Causal Transformers without Position Encoding

Length Generalization of Causal Transformers without Position Encoding

18 April 2024

Xuanjing Huang

Papers citing "Length Generalization of Causal Transformers without Position Encoding"

5 / 5 papers shown

Title
Round and Round We Go! What makes Rotary Positional Encodings useful? Federico Barbero Alex Vitvitskyi Christos Perivolaropoulos Razvan Pascanu Petar Velickovic 75 16 0 08 Oct 2024
AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark Wenhao Chai Enxin Song Y. Du Chenlin Meng Vashisht Madhavan Omer Bar-Tal Jeng-Neng Hwang Saining Xie Christopher D. Manning 3DV 77 25 0 04 Oct 2024
Mitigate Position Bias in Large Language Models via Scaling a Single Dimension Yijiong Yu Huiqiang Jiang Xufang Luo Qianhui Wu Chin-Yew Lin Dongsheng Li Yuqing Yang Yongfeng Huang L. Qiu 40 9 0 04 Jun 2024
Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddings Ta-Chung Chi Ting-Han Fan Li-Wei Chen Alexander I. Rudnicky Peter J. Ramadge VLM MILM 52 12 0 23 May 2023
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press Noah A. Smith M. Lewis 245 695 0 27 Aug 2021