Resonance RoPE: Improving Context Length Generalization of Large
Language Models

Resonance RoPE: Improving Context Length Generalization of Large Language Models

29 February 2024

Mehdi Rezagholizadeh

Papers citing "Resonance RoPE: Improving Context Length Generalization of Large Language Models"

11 / 11 papers shown

Title
LongMamba: Enhancing Mamba's Long Context Capabilities via Training-Free Receptive Field Enlargement Zhifan Ye Kejing Xia Yonggan Fu Xin Dong Jihoon Hong Xiangchi Yuan Shizhe Diao Jan Kautz Pavlo Molchanov Yingyan Lin Mamba 45 3 0 22 Apr 2025
Of All StrIPEs: Investigating Structure-informed Positional Encoding for Efficient Music Generation Manvi Agarwal Changhong Wang Gaël Richard 27 0 0 07 Apr 2025
LLMs are Also Effective Embedding Models: An In-depth Overview Chongyang Tao Tao Shen Shen Gao Junshuo Zhang Zhen Li Zhengwei Tao Shuai Ma 71 7 0 17 Dec 2024
When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Training Haonan Wang Qian Liu Chao Du Tongyao Zhu Cunxiao Du Kenji Kawaguchi Tianyu Pang 94 6 0 20 Nov 2024
DAPE V2: Process Attention Score as Feature Map for Length Extrapolation Chuanyang Zheng Yihang Gao Han Shi Jing Xiong Jiankai Sun ... Xiaozhe Ren Michael Ng Xin Jiang Zhenguo Li Yu Li 31 2 0 07 Oct 2024
A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts Suyu Ge Xihui Lin Yunan Zhang Jiawei Han Hao Peng 31 4 0 02 Oct 2024
LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Models Zhiyuan Hu Yuliang Liu Jinman Zhao Suyuchen Wang Yan Wang ... Qing Gu Anh Tuan Luu See-Kiong Ng Zhiwei Jiang Bryan Hooi 55 11 0 31 Aug 2024
LongEmbed: Extending Embedding Models for Long Context Retrieval Dawei Zhu Liang Wang Nan Yang Yifan Song Wenhao Wu Furu Wei Sujian Li RALM 40 21 0 18 Apr 2024
PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training Dawei Zhu Nan Yang Liang Wang Yifan Song Wenhao Wu Furu Wei Sujian Li 55 77 0 19 Sep 2023
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press Noah A. Smith M. Lewis 245 695 0 27 Aug 2021
ZeRO-Offload: Democratizing Billion-Scale Model Training Jie Ren Samyam Rajbhandari Reza Yazdani Aminabadi Olatunji Ruwase Shuangyang Yang Minjia Zhang Dong Li Yuxiong He MoE 160 413 0 18 Jan 2021