LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models

21 September 2023

Zhijian Liu

Song Han

Jiaya Jia

ArXiv PDF HTML

Papers citing "LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models"

50 / 122 papers shown

Title
VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computation Shiwei Wu Joya Chen Kevin Qinghong Lin Qimeng Wang Yan Gao Qianli Xu Tong Bill Xu Yao Hu Enhong Chen Mike Zheng Shou VLM 45 12 0 29 Aug 2024
LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs Yushi Bai Jiajie Zhang Xin Lv Linzhi Zheng Siqi Zhu Lei Hou Yuxiao Dong Jie Tang Juanzi Li VGen LLMAG ALM 34 37 0 13 Aug 2024
MINI-SEQUENCE TRANSFORMER: Optimizing Intermediate Memory for Long Sequences Training Cheng Luo Jiawei Zhao Zhuoming Chen Beidi Chen A. Anandkumar 18 3 0 22 Jul 2024
LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inference Qichen Fu Minsik Cho Thomas Merth Sachin Mehta Mohammad Rastegari Mahyar Najibi 33 25 0 19 Jul 2024
ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities Peng-Tao Xu Wei Ping Xianchao Wu Zihan Liu M. Shoeybi Mohammad Shoeybi Bryan Catanzaro RALM 44 14 0 19 Jul 2024
DreamStory: Open-Domain Story Visualization by LLM-Guided Multi-Subject Consistent Diffusion Huiguo He Huan Yang Zixi Tuo Yuan Zhou Qiuyue Wang Yuhang Zhang Zeyu Liu Wenhao Huang Hongyang Chao Jian Yin DiffM VGen 52 12 0 17 Jul 2024
Human-like Episodic Memory for Infinite Context LLMs Z. Fountas Martin A Benfeghoul Adnan Oomerjee Fenia Christopoulou Gerasimos Lampouras Haitham Bou-Ammar Jun Wang 29 18 0 12 Jul 2024
SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers Shraman Pramanick Rama Chellappa Subhashini Venugopalan 43 13 0 12 Jul 2024
SBoRA: Low-Rank Adaptation with Regional Weight Updates L. Po Yuyang Liu Haoxuan Wu Tianqi Zhang W. Yu Zeyu Jiang Kun Li 33 0 0 07 Jul 2024
MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizations Yubo Ma Yuhang Zang Liangyu Chen Meiqi Chen Yizhu Jiao ... Liangming Pan Yu-Gang Jiang Jiaqi Wang Yixin Cao Aixin Sun ELM RALM VLM 24 23 0 01 Jul 2024
Increasing Model Capacity for Free: A Simple Strategy for Parameter Efficient Fine-tuning Haobo Song Hao Zhao Soumajit Majumder Tao Lin 23 3 0 01 Jul 2024
Revisiting Random Walks for Learning on Graphs Jinwoo Kim Olga Zaghen Ayhan Suleymanzade Youngmin Ryou Seunghoon Hong 54 0 0 01 Jul 2024
UIO-LLMs: Unbiased Incremental Optimization for Long-Context LLMs Wenhao Li Mingbao Lin Yunshan Zhong Shuicheng Yan Rongrong Ji 28 0 0 26 Jun 2024
Evaluating the Efficacy of Foundational Models: Advancing Benchmarking Practices to Enhance Fine-Tuning Decision-Making O. Amujo S. Yang 22 0 0 25 Jun 2024
Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformers Chao Lou Zixia Jia Zilong Zheng Kewei Tu ODL 31 18 0 24 Jun 2024
DeciMamba: Exploring the Length Extrapolation Potential of Mamba Assaf Ben-Kish Itamar Zimerman Shady Abu Hussein Nadav Cohen Amir Globerson Lior Wolf Raja Giryes Mamba 67 13 0 20 Jun 2024
Efficient Prompting for LLM-based Generative Internet of Things Bin Xiao B. Kantarci Jiawen Kang Dusit Niyato Mohsen Guizani 28 16 0 14 Jun 2024
3D-RPE: Enhancing Long-Context Modeling Through 3D Rotary Position Encoding Xindian Ma Wenyuan Liu Peng Zhang Nan Xu 29 3 0 14 Jun 2024
Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling Liliang Ren Yang Liu Yadong Lu Yelong Shen Chen Liang Weizhu Chen Mamba 64 55 0 11 Jun 2024
SinkLoRA: Enhanced Efficiency and Chat Capabilities for Long-Context Large Language Models Hengyu Zhang RALM 20 2 0 09 Jun 2024
Quest: Query-centric Data Synthesis Approach for Long-context Scaling of Large Language Model Chaochen Gao Xing Wu Qingfang Fu Songlin Hu SyDa 27 5 0 30 May 2024
TAGA: Text-Attributed Graph Self-Supervised Learning by Synergizing Graph and Text Mutual Transformations Zhengwu Zhang Yuntong Hu Bo Pan Chen Ling Liang Zhao 31 2 0 27 May 2024
Compressing Lengthy Context With UltraGist Peitian Zhang Zheng Liu Shitao Xiao Ninglu Shao Qiwei Ye Zhicheng Dou 19 4 0 26 May 2024
HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models Bernal Jiménez Gutiérrez Yiheng Shu Yu Gu Michihiro Yasunaga Yu-Chuan Su RALM CLL 54 31 0 23 May 2024
From Generalist to Specialist: Improving Large Language Models for Medical Physics Using ARCoT Jace Grandinetti R. Mcbeth AI4CE LRM LM&MA 27 0 0 17 May 2024
Long Context Alignment with Short Instructions and Synthesized Positions Wenhao Wu Yizhong Wang Yao Fu Xiang Yue Dawei Zhu Sujian Li SyDa 35 18 0 07 May 2024
Gaussian Stochastic Weight Averaging for Bayesian Low-Rank Adaptation of Large Language Models Emre Onal Klemens Flöge Emma Caldwell A. Sheverdin Vincent Fortuin UQCV BDL 32 9 0 06 May 2024
From Persona to Personalization: A Survey on Role-Playing Language Agents Jiangjie Chen Xintao Wang Rui Xu Siyu Yuan Yikai Zhang ... Caiyu Hu Siye Wu Scott Ren Ziquan Fu Yanghua Xiao 50 72 0 28 Apr 2024
LLoCO: Learning Long Contexts Offline Sijun Tan Xiuyu Li Shishir G. Patil Ziyang Wu Tianjun Zhang Kurt Keutzer Joseph E. Gonzalez Raluca A. Popa RALM OffRL LLMAG 38 6 0 11 Apr 2024
Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention Tsendsuren Munkhdalai Manaal Faruqui Siddharth Gopal LRM LLMAG CLL 81 101 0 10 Apr 2024
Long-context LLMs Struggle with Long In-context Learning Tianle Li Ge Zhang Quy Duc Do Xiang Yue Wenhu Chen 42 152 0 02 Apr 2024
MTLoRA: A Low-Rank Adaptation Approach for Efficient Multi-Task Learning Ahmed A. Agiza Marina Neseem Sherief Reda MoE 30 12 0 29 Mar 2024
MLDT: Multi-Level Decomposition for Complex Long-Horizon Robotic Task Planning with Open-Source Large Language Model Yike Wu Jiatao Zhang Nan Hu LanLing Tang Guilin Qi Jun Shao Jie Ren Wei Song 57 10 0 27 Mar 2024
Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning Hao Shao Shengju Qian Han Xiao Guanglu Song Zhuofan Zong Letian Wang Yu Liu Hongsheng Li VGen LRM MLLM 58 35 0 25 Mar 2024
CodeS: Natural Language to Code Repository via Multi-Layer Sketch Daoguang Zan Ailun Yu Wei Liu Dong Chen Bo Shen ... Bei Guan Zhiguang Yang Yongji Wang Qianxiang Wang Li-zhen Cui 18 14 0 25 Mar 2024
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey Zeyu Han Chao Gao Jinyang Liu Jeff Zhang Sai Qian Zhang 136 301 0 21 Mar 2024
StreamingDialogue: Prolonged Dialogue Learning via Long Context Compression with Minimal Losses Jia-Nan Li Quan Tu Cunli Mao Zhengtao Yu Ji-Rong Wen Rui Yan OffRL 21 3 0 13 Mar 2024
Matrix-Transformation Based Low-Rank Adaptation (MTLoRA): A Brain-Inspired Method for Parameter-Efficient Fine-Tuning Yao Liang Yuwei Wang Yang Li Yi Zeng 37 0 0 12 Mar 2024
ResAdapter: Domain Consistent Resolution Adapter for Diffusion Models Jiaxiang Cheng Pan Xie Xin Xia Jiashi Li Jie Wu Yuxi Ren Huixia Li Xuefeng Xiao Min Zheng Lean Fu 33 12 0 04 Mar 2024
DropBP: Accelerating Fine-Tuning of Large Language Models by Dropping Backward Propagation Sunghyeon Woo Baeseong Park Byeongwook Kim Minjung Jo S. Kwon Dongsuk Jeon Dongsoo Lee 54 2 0 27 Feb 2024
Me LLaMA: Foundation Large Language Models for Medical Applications Qianqian Xie Qingyu Chen Aokun Chen C.A.I. Peng Yan Hu ... Huan He Lucila Ohno-Machido Yonghui Wu Hua Xu Jiang Bian LM&MA AI4MH 70 3 0 20 Feb 2024
BitDelta: Your Fine-Tune May Only Be Worth One Bit James Liu Guangxuan Xiao Kai Li Jason D. Lee Song Han Tri Dao Tianle Cai 23 20 0 15 Feb 2024
A Human-Inspired Reading Agent with Gist Memory of Very Long Contexts Kuang-Huei Lee Xinyun Chen Hiroki Furuta John F. Canny Ian S. Fischer RALM 53 29 0 15 Feb 2024
The Sound of Healthcare: Improving Medical Transcription ASR Accuracy with Large Language Models Ayo Adedeji Sarita Joshi Brendan Doohan LM&MA 22 14 0 12 Feb 2024
UniMem: Towards a Unified View of Long-Context Large Language Models Junjie Fang Likai Tang Hongzhe Bi Yujia Qin Si Sun ... Xiaodong Shi Sen Song Yankai Lin Zhiyuan Liu Maosong Sun 11 3 0 05 Feb 2024
KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization Coleman Hooper Sehoon Kim Hiva Mohammadzadeh Michael W. Mahoney Y. Shao Kurt Keutzer A. Gholami MQ 12 171 0 31 Jan 2024
In-context Learning with Retrieved Demonstrations for Language Models: A Survey an Luo Xin Xu Yue Liu Panupong Pasupat Mehran Kazemi RALM 24 54 0 21 Jan 2024
With Greater Text Comes Greater Necessity: Inference-Time Training Helps Long Text Generation Y. Wang D. Ma D. Cai RALM 28 17 0 21 Jan 2024
CaraServe: CPU-Assisted and Rank-Aware LoRA Serving for Generative LLM Inference Suyi Li Hanfeng Lu Tianyuan Wu Minchen Yu Qizhen Weng Xusheng Chen Yizhou Shan Binhang Yuan Wei Wang 48 11 0 20 Jan 2024
Flexibly Scaling Large Language Models Contexts Through Extensible Tokenization Ninglu Shao Shitao Xiao Zheng Liu Peitian Zhang 18 4 0 15 Jan 2024