LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning

2 January 2024

Chia-Yuan Chang

Papers citing "LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning"

50 / 69 papers shown

Title
Leveraging State Space Models in Long Range Genomics Matvei Popov Aymen Kallala Anirudha Ramesh Narimane Hennouni Shivesh Khaitan Rick Gentry Alain-Sam Cohen Mamba 60 0 0 07 Apr 2025
LongDiff: Training-Free Long Video Generation in One Go Zhuoling Li Hossein Rahmani Qiuhong Ke J. Liu DiffM VGen VLM 56 0 0 23 Mar 2025
SkyLadder: Better and Faster Pretraining via Context Window Scheduling Tongyao Zhu Qian Liu Haonan Wang Shiqi Chen Xiangming Gu Tianyu Pang Min-Yen Kan 36 0 0 19 Mar 2025
A Survey of Large Language Model Empowered Agents for Recommendation and Search: Towards Next-Generation Information Retrieval Yu Zhang Shutong Qiao Jiaqi Zhang Tzu-Heng Lin Chen Gao Y. Li LM&Ro LM&MA 87 1 0 07 Mar 2025
Layer-Specific Scaling of Positional Encodings for Superior Long-Context Modeling Zhenghua Wang Yiran Ding Changze Lv Zhibo Xu Tianlong Li Tianyuan Shi Xiaoqing Zheng Xuanjing Huang 35 0 0 06 Mar 2025
LongAttn: Selecting Long-context Training Data via Token-level Attention Longyun Wu Dawei Zhu Guangxiang Zhao Zhuocheng Yu Junfeng Ran Xiangyu Wong Lin Sun Sujian Li 36 0 0 24 Feb 2025
WildLong: Synthesizing Realistic Long-Context Instruction Data at Scale Jiaxi Li Xingxing Zhang Xun Wang Xiaolong Huang Li Dong Liang Wang Si-Qing Chen Wei Lu Furu Wei SyDa 91 0 0 23 Feb 2025
MMRAG: Multi-Mode Retrieval-Augmented Generation with Large Language Models for Biomedical In-Context Learning Zaifu Zhan J. Wang Shuang Zhou Jiawen Deng Rui Zhang 40 4 0 21 Feb 2025
CalibQuant: 1-Bit KV Cache Quantization for Multimodal LLMs Zeliang Zhang Yifan Zhu Susan Liang Zhiyuan Wang Jiani Liu ... Mingjie Zhao Chenliang Xu Kun Wan Wentian Zhao Wentian Zhao VLM MQ 41 0 0 15 Feb 2025
InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on a Single GPU Heejun Lee G. Park Jaduk Suh Sung Ju Hwang 82 1 0 13 Feb 2025
SEAL: Scaling to Emphasize Attention for Long-Context Retrieval Changhun Lee Jun-gyu Jin Younghyun Cho Eunhyeok Park LRM 46 0 0 28 Jan 2025
NExtLong: Toward Effective Long-Context Training without Long Documents Chaochen Gao Xing Wu Zijia Lin Debing Zhang Songlin Hu SyDa 64 1 0 22 Jan 2025
Leveraging Fine-Tuned Retrieval-Augmented Generation with Long-Context Support: For 3GPP Standards Omar Erak Nouf Alabbasi Omar Alhussein Ismail Lotfi Amr Hussein Sami Muhaidat Merouane Debbah RALM 76 5 0 17 Jan 2025
LIFT: Improving Long Context Understanding Through Long Input Fine-Tuning Yansheng Mao Jiaqi Li Fanxu Meng Jing Xiong Zilong Zheng Muhan Zhang LLMAG RALM 92 1 0 18 Dec 2024
LLMs are Also Effective Embedding Models: An In-depth Overview Chongyang Tao Tao Shen Shen Gao Junshuo Zhang Zhen Li Zhengwei Tao Shuai Ma 68 7 0 17 Dec 2024
Unified Parameter-Efficient Unlearning for LLMs Chenlu Ding Jiancan Wu Yancheng Yuan Jinda Lu Kai Zhang Alex Su Xiang Wang Xiangnan He MU KELM 97 6 0 30 Nov 2024
When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Training Haonan Wang Qian Liu Chao Du Tongyao Zhu Cunxiao Du Kenji Kawaguchi Tianyu Pang 88 5 0 20 Nov 2024
AIDBench: A benchmark for evaluating the authorship identification capability of large language models Zichen Wen Dadi Guo Huishuai Zhang 67 0 0 20 Nov 2024
TeleOracle: Fine-Tuned Retrieval-Augmented Generation with Long-Context Support for Network Nouf Alabbasi Omar Erak Omar Alhussein Ismail Lotfi Sami Muhaidat Merouane Debbah RALM 76 0 0 04 Nov 2024
LOGO -- Long cOntext aliGnment via efficient preference Optimization Zecheng Tang Zechen Sun Juntao Li Qiaoming Zhu Min Zhang 27 0 0 24 Oct 2024
Mesa-Extrapolation: A Weave Position Encoding Method for Enhanced Extrapolation in LLMs Xin Ma Yang Liu J. Liu Xiaoxu Ma 11 1 0 21 Oct 2024
An Evolved Universal Transformer Memory Edoardo Cetin Qi Sun Tianyu Zhao Yujin Tang 63 0 0 17 Oct 2024
On the token distance modeling ability of higher RoPE attention dimension Xiangyu Hong Che Jiang Biqing Qi Fandong Meng Mo Yu Bowen Zhou Jie Zhou 31 4 0 11 Oct 2024
Stuffed Mamba: State Collapse and State Capacity of RNN-Based Long-Context Modeling Yingfa Chen Xinrong Zhang Shengding Hu Xu Han Zhiyuan Liu Maosong Sun Mamba 51 2 0 09 Oct 2024
DAPE V2: Process Attention Score as Feature Map for Length Extrapolation Chuanyang Zheng Yihang Gao Han Shi Jing Xiong Jiankai Sun ... Xiaozhe Ren Michael Ng Xin Jiang Zhenguo Li Yu Li 26 1 0 07 Oct 2024
How to Train Long-Context Language Models (Effectively) Tianyu Gao Alexander Wettig Howard Yen Danqi Chen RALM 69 37 0 03 Oct 2024
ImageFolder: Autoregressive Image Generation with Folded Tokens Xiang Li Kai Qiu Hao Chen Jason Kuen Jiuxiang Gu Bhiksha Raj Zhe-nan Lin VLM 34 17 0 02 Oct 2024
Role-RL: Online Long-Context Processing with Role Reinforcement Learning for Distinct LLMs in Their Optimal Roles Lewei He Tianyu Shi Pengran Huang Bingzhi Chen Qianglong Chen Jiahui Pan OffRL 33 0 0 26 Sep 2024
Towards LifeSpan Cognitive Systems Yu Wang Chi Han Tongtong Wu Xiaoxin He Wangchunshu Zhou ... Zexue He Wei Wang Gholamreza Haffari Heng Ji Julian McAuley KELM CLL 88 1 0 20 Sep 2024
InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference Xiurui Pan Endian Li Qiao Li Shengwen Liang Yizhou Shan Ke Zhou Yingwei Luo Xiaolin Wang Jie Zhang 28 10 0 08 Sep 2024
What are the Essential Factors in Crafting Effective Long Context Multi-Hop Instruction Datasets? Insights and Best Practices Zhi Chen Qiguang Chen Libo Qin Qipeng Guo Haijun Lv Yicheng Zou Wanxiang Che Hang Yan Kai Chen Dahua Lin SyDa 38 4 0 03 Sep 2024
LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Models Zhiyuan Hu Yuliang Liu Jinman Zhao Suyuchen Wang Yan Wang ... Qing Gu Anh Tuan Luu See-Kiong Ng Zhiwei Jiang Bryan Hooi 50 11 0 31 Aug 2024
Agentic Retrieval-Augmented Generation for Time Series Analysis Chidaksh Ravuru Sagar Srinivas Sakhinana Venkataramana Runkana AI4TS 18 5 0 18 Aug 2024
LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs Yushi Bai Jiajie Zhang Xin Lv Linzhi Zheng Siqi Zhu Lei Hou Yuxiao Dong Jie Tang Juanzi Li VGen LLMAG ALM 34 37 0 13 Aug 2024
Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption Shi Luohe Hongyi Zhang Yao Yao Z. Li Zhao Hai 31 31 0 25 Jul 2024
Grammar-based Game Description Generation using Large Language Models Tsunehiko Tanaka Edgar Simo-Serra 48 2 0 24 Jul 2024
ReAttention: Training-Free Infinite Context with Finite Attention Scope Xiaoran Liu Ruixiao Li Yuerong Song Zhigeng Liu Kai Lv Hang Yan Hang Yan Linlin Li Qun Liu Xipeng Qiu LLMAG 25 1 0 21 Jul 2024
From Artificial Needles to Real Haystacks: Improving Retrieval Capabilities in LLMs by Finetuning on Synthetic Data Zheyang Xiong Vasilis Papageorgiou Kangwook Lee Dimitris Papailiopoulos SyDa RALM 24 11 0 27 Jun 2024
SimCE: Simplifying Cross-Entropy Loss for Collaborative Filtering Xiaodong Yang Huiyuan Chen Yuchen Yan Yuxin Tang Yuying Zhao Eric Xu Yiwei Cai Hanghang Tong 13 3 0 23 Jun 2024
LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMs Ziyan Jiang Xueguang Ma Wenhu Chen RALM 41 47 0 21 Jun 2024
MedOdyssey: A Medical Domain Benchmark for Long Context Evaluation Up to 200K Tokens Yongqi Fan Hongli Sun Kui Xue Xiaofan Zhang Shaoting Zhang Tong Ruan 34 0 0 21 Jun 2024
From Text to Life: On the Reciprocal Relationship between Artificial Life and Large Language Models Eleni Nisioti Claire Glanois Elias Najarro Andrew Dai Elliot Meyerson J. Pedersen Laetitia Teodorescu Conor F. Hayes Shyam Sudhakaran Sebastian Risi AI4CE LM&Ro 35 2 0 14 Jun 2024
Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling Liliang Ren Yang Liu Yadong Lu Yelong Shen Chen Liang Weizhu Chen Mamba 64 55 0 11 Jun 2024
LoCoCo: Dropping In Convolutions for Long Context Compression Ruisi Cai Yuandong Tian Zhangyang Wang Beidi Chen 33 9 0 08 Jun 2024
From Text to Pixel: Advancing Long-Context Understanding in MLLMs Yujie Lu Xiujun Li Tsu-jui Fu Miguel P. Eckstein William Yang Wang VLM 37 2 0 23 May 2024
Position: Leverage Foundational Models for Black-Box Optimization Xingyou Song Yingtao Tian Robert Tjarko Lange Chansoo Lee Yujin Tang Yutian Chen 38 5 0 06 May 2024
Towards Incremental Learning in Large Language Models: A Critical Review M. Jovanovic Peter Voss ELM CLL KELM 26 5 0 28 Apr 2024
SnapKV: LLM Knows What You are Looking for Before Generation Yuhong Li Yingbing Huang Bowen Yang Bharat Venkitesh Acyr F. Locatelli Hanchen Ye Tianle Cai Patrick Lewis Deming Chen VLM 75 148 0 22 Apr 2024
Length Generalization of Causal Transformers without Position Encoding Jie Wang Tao Ji Yuanbin Wu Hang Yan Tao Gui Qi Zhang Xuanjing Huang Xiaoling Wang VLM 31 15 0 18 Apr 2024
LongEmbed: Extending Embedding Models for Long Context Retrieval Dawei Zhu Liang Wang Nan Yang Yifan Song Wenhao Wu Furu Wei Sujian Li RALM 35 20 0 18 Apr 2024