LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models

21 September 2023

Zhijian Liu

Song Han

Jiaya Jia

ArXiv PDF HTML

Papers citing "LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models"

50 / 122 papers shown

Title
Retrieval Augmented Generation Evaluation for Health Documents Mario Ceresa Lorenzo Bertolini Valentin Comte Nicholas Spadaro Barbara Raffael ... Sergio Consoli Amalia Muñoz Piñeiro Alex Patak Maddalena Querci Tobias Wiesenthal RALM 3DV 31 0 1 07 May 2025
VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Model Zuwei Long Yunhang Shen Chaoyou Fu Heting Gao Lijiang Li ... Jinlong Peng Haoyu Cao Ke Li R. Ji Xing Sun 30 0 0 06 May 2025
FreqKV: Frequency Domain Key-Value Compression for Efficient Context Window Extension Jushi Kai Boyi Zeng Y. Wang Haoli Bai Bo Jiang Zhouhan Lin 32 0 0 01 May 2025
Graph-Based Spectral Decomposition for Parameter Coordination in Language Model Fine-Tuning Hanlu Zhang Yumeng Ma Shuo Wang Guiran Liu B. Zhu 35 0 0 28 Apr 2025
SWE-Synth: Synthesizing Verifiable Bug-Fix Data to Enable Large Language Models in Resolving Real-World Bugs Minh V.T. Pham Huy N. Phan Hoang N. Phan Cuong Le Chi T. Nguyen Nghi D. Q. Bui SyDa 24 0 0 20 Apr 2025
Aspect-Based Summarization with Self-Aspect Retrieval Enhanced Generation Yichao Feng Shuai Zhao Y. Li Luwei Xiao Xiaobao Wu Anh Tuan Luu RALM 23 0 0 17 Apr 2025
M-DocSum: Do LVLMs Genuinely Comprehend Interleaved Image-Text in Document Summarization? Haolong Yan Kaijun Tan Yeqing Shen Xin Huang Zheng Ge Xiangyu Zhang Si Li Daxin Jiang VLM 35 0 0 27 Mar 2025
Long-Context Autoregressive Video Modeling with Next-Frame Prediction Yuchao Gu Weijia Mao Mike Zheng Shou VGen 71 1 0 25 Mar 2025
Temporal Action Detection Model Compression by Progressive Block Drop Xiaoyong Chen Yong Guo Jiaming Liang Sitong Zhuang Runhao Zeng Xiping Hu 43 0 0 21 Mar 2025
VisRL: Intention-Driven Visual Perception via Reinforced Reasoning Zhangquan Chen Xufang Luo Dongsheng Li OffRL LRM 62 3 0 10 Mar 2025
LongAttn: Selecting Long-context Training Data via Token-level Attention Longyun Wu Dawei Zhu Guangxiang Zhao Zhuocheng Yu Junfeng Ran Xiangyu Wong Lin Sun Sujian Li 36 0 0 24 Feb 2025
WildLong: Synthesizing Realistic Long-Context Instruction Data at Scale Jiaxi Li Xingxing Zhang Xun Wang Xiaolong Huang Li Dong Liang Wang Si-Qing Chen Wei Lu Furu Wei SyDa 60 0 0 23 Feb 2025
SimPER: A Minimalist Approach to Preference Alignment without Hyperparameters Teng Xiao Yige Yuan Z. Chen Mingxiao Li Shangsong Liang Z. Ren V. Honavar 93 5 0 21 Feb 2025
ELITR-Bench: A Meeting Assistant Benchmark for Long-Context Language Models Thibaut Thonet Jos Rozen Laurent Besacier RALM 129 2 0 20 Jan 2025
Rethinking Addressing in Language Models via Contexualized Equivariant Positional Encoding Jiajun Zhu Peihao Wang Ruisi Cai Jason D. Lee Pan Li Z. Wang KELM 36 1 0 03 Jan 2025
LIFT: Improving Long Context Understanding Through Long Input Fine-Tuning Yansheng Mao Jiaqi Li Fanxu Meng Jing Xiong Zilong Zheng Muhan Zhang LLMAG RALM 90 1 0 18 Dec 2024
Expansion Span: Combining Fading Memory and Retrieval in Hybrid State Space Models Elvis Nunez L. Zancato Benjamin Bowman Aditya Golatkar W. Xia Stefano Soatto 73 2 0 17 Dec 2024
LLMs are Also Effective Embedding Models: An In-depth Overview Chongyang Tao Tao Shen Shen Gao Junshuo Zhang Zhen Li Zhengwei Tao Shuai Ma 66 7 0 17 Dec 2024
Lyra: An Efficient and Speech-Centric Framework for Omni-Cognition Zhisheng Zhong Chengyao Wang Yuqi Liu Senqiao Yang Longxiang Tang ... Shaozuo Yu Sitong Wu Eric Lo Shu-Lin Liu Jiaya Jia AuLLM 100 6 0 12 Dec 2024
Quantized Delta Weight Is Safety Keeper Yule Liu Zhen Sun Xinlei He Xinyi Huang 72 2 0 29 Nov 2024
Learning from "Silly" Questions Improves Large Language Models, But Only Slightly Tingyuan Zhu Shudong Liu Yidong Wang Derek F. Wong Han Yu T. Shinozaki Jindong Wang ALM LRM 72 0 0 21 Nov 2024
Next-Generation Phishing: How LLM Agents Empower Cyber Attackers Khalifa Afane Wenqi Wei Ying Mao Junaid Farooq Juntao Chen 70 2 0 21 Nov 2024
When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Training Haonan Wang Qian Liu Chao Du Tongyao Zhu Cunxiao Du Kenji Kawaguchi Tianyu Pang 82 5 0 20 Nov 2024
Squeezed Attention: Accelerating Long Context Length LLM Inference Coleman Hooper Sehoon Kim Hiva Mohammadzadeh Monishwaran Maheswaran June Paik Michael W. Mahoney K. K. Amir Gholami 48 9 0 14 Nov 2024
What is Wrong with Perplexity for Long-context Language Modeling? Lizhe Fang Yifei Wang Zhaoyang Liu Chenheng Zhang Stefanie Jegelka Jinyang Gao Bolin Ding Yisen Wang 51 4 0 31 Oct 2024
LOGO -- Long cOntext aliGnment via efficient preference Optimization Zecheng Tang Zechen Sun Juntao Li Qiaoming Zhu Min Zhang 22 0 0 24 Oct 2024
Mitigating Forgetting in LLM Supervised Fine-Tuning and Preference Learning H. Fernando Han Shen Parikshit Ram Yi Zhou Horst Samulowitz Nathalie Baracaldo Tianyi Chen CLL 50 2 0 20 Oct 2024
Rethinking Transformer for Long Contextual Histopathology Whole Slide Image Analysis Honglin Li Yunlong Zhang Pingyi Chen Zhongyi Shui Chenglu Zhu Lin Yang MedIm 29 4 0 18 Oct 2024
From Isolated Conversations to Hierarchical Schemas: Dynamic Tree Memory Representation for LLMs Alireza Rezazadeh Zichao Li Wei Wei Yujia Bao 30 4 0 17 Oct 2024
LR-SQL: A Supervised Fine-Tuning Method for Text2SQL Tasks under Low-Resource Scenarios Wen Wuzhenghong Zhang Yongpan Pan Su Sun Yuwei Lu Pengwei Ding Cheng 17 0 0 15 Oct 2024
Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Free Ziyue Li Tianyi Zhou MoE 55 16 0 14 Oct 2024
FltLM: An Intergrated Long-Context Large Language Model for Effective Context Filtering and Understanding Jingyang Deng Zhengyang Shen Boyang Wang Lixin Su Suqi Cheng Ying Nie Junfeng Wang Dawei Yin Jinwen Ma 15 1 0 09 Oct 2024
HALL-E: Hierarchical Neural Codec Language Model for Minute-Long Zero-Shot Text-to-Speech Synthesis Yuto Nishimura Takumi Hirose Masanari Ohi Hideki Nakayama Nakamasa Inoue VLM 26 1 0 06 Oct 2024
BlockFound: Customized blockchain foundation model for anomaly detection Jiahao Yu Xian Wu Hao Liu Wenbo Guo Xinyu Xing 14 1 0 05 Oct 2024
Geometric Collaborative Filtering with Convergence Hisham Husain Julien Monteil FedML 23 5 0 04 Oct 2024
CalliffusionV2: Personalized Natural Calligraphy Generation with Flexible Multi-modal Control Qisheng Liao Liang Li Yulang Fei Gus Xia DiffM VLM 13 0 0 03 Oct 2024
UncertaintyRAG: Span-Level Uncertainty Enhanced Long-Context Modeling for Retrieval-Augmented Generation Zixuan Li Jing Xiong Fanghua Ye Chuanyang Zheng Xun Wu ... Xiaodan Liang Chengming Li Zhenan Sun Lingpeng Kong Ngai Wong RALM UQLM 27 0 0 03 Oct 2024
SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration Jintao Zhang Jia wei Pengle Zhang Jun-Jie Zhu Jun Zhu Jianfei Chen VLM MQ 74 18 0 03 Oct 2024
How to Train Long-Context Language Models (Effectively) Tianyu Gao Alexander Wettig Howard Yen Danqi Chen RALM 62 36 0 03 Oct 2024
Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads on Consumer-Grade Devices Yuxiang Huang Binhang Yuan Xu Han Chaojun Xiao Zhiyuan Liu RALM 69 1 0 02 Oct 2024
FlashMask: Efficient and Rich Mask Extension of FlashAttention Guoxia Wang Jinle Zeng Xiyuan Xiao Siming Wu Jiabin Yang Lujing Zheng Zeyu Chen Jiang Bian Dianhai Yu Haifeng Wang 36 2 0 02 Oct 2024
LoRKD: Low-Rank Knowledge Decomposition for Medical Foundation Models Haolin Li Yuhang Zhou Ziheng Zhao Siyuan Du Jiangchao Yao Weidi Xie Ya Zhang Yanfeng Wang 29 1 0 29 Sep 2024
HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models Haoran Que Feiyu Duan Liqun He Yutao Mou Wangchunshu Zhou ... Ge Zhang Junran Peng Zhaoxiang Zhang Songyang Zhang Kai Chen LM&MA ELM VLM 43 11 0 24 Sep 2024
Uncovering Latent Chain of Thought Vectors in Language Models Jason Zhang Scott Viteri LLMSV LRM 33 1 0 21 Sep 2024
Towards LifeSpan Cognitive Systems Yu Wang Chi Han Tongtong Wu Xiaoxin He Wangchunshu Zhou ... Zexue He Wei Wang Gholamreza Haffari Heng Ji Julian McAuley KELM CLL 83 1 0 20 Sep 2024
A Controlled Study on Long Context Extension and Generalization in LLMs Yi Lu Jing Nathan Yan Songlin Yang Justin T. Chiu Siyu Ren Fei Yuan Wenting Zhao Zhiyong Wu Alexander M. Rush 28 9 0 18 Sep 2024
Schrodinger's Memory: Large Language Models Wei Wang Qing Li 19 1 0 16 Sep 2024
E2LLM: Encoder Elongated Large Language Models for Long-Context Understanding and Reasoning Zihan Liao Jun Wang Hang Yu Lingxiao Wei Jianguo Li Jun Wang Wei Zhang 19 2 0 10 Sep 2024
What are the Essential Factors in Crafting Effective Long Context Multi-Hop Instruction Datasets? Insights and Best Practices Zhi Chen Qiguang Chen Libo Qin Qipeng Guo Haijun Lv Yicheng Zou Wanxiang Che Hang Yan Kai Chen Dahua Lin SyDa 38 4 0 03 Sep 2024
VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computation Shiwei Wu Joya Chen Kevin Qinghong Lin Qimeng Wang Yan Gao Qianli Xu Tong Bill Xu Yao Hu Enhong Chen Mike Zheng Shou VLM 37 0 0 29 Aug 2024