LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models

21 September 2023

Zhijian Liu

Song Han

Jiaya Jia

ArXiv PDF HTML

Papers citing "LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models"

22 / 122 papers shown

Title
Training With "Paraphrasing the Original Text'' Improves Long-Context Performance Yijiong Yu RALM 10 3 0 18 Dec 2023
MagicScroll: Nontypical Aspect-Ratio Image Generation for Visual Storytelling via Multi-Layered Semantic-Aware Denoising Bingyuan Wang Hengyu Meng Zeyu Cai Lanjiong Li Yue Ma Qifeng Chen Zeyu Wang DiffM 23 3 0 18 Dec 2023
Extending Context Window of Large Language Models via Semantic Compression WeiZhi Fei Xueyan Niu Pingyi Zhou Lu Hou Bo Bai Lei Deng Wei Han 26 26 0 15 Dec 2023
The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning Bill Yuchen Lin Abhilasha Ravichander Ximing Lu Nouha Dziri Melanie Sclar Khyathi Raghavi Chandu Chandra Bhagavatula Yejin Choi 17 163 0 04 Dec 2023
NLEBench+NorGLM: A Comprehensive Empirical Analysis and Benchmark Dataset for Generative Language Models in Norwegian Peng Liu Lemei Zhang Terje Nissen Farup Even W. Lauvrak Jon Espen Ingvaldsen Simen Eide J. Gulla Zhirong Yang ELM 16 6 0 03 Dec 2023
USat: A Unified Self-Supervised Encoder for Multi-Sensor Satellite Imagery Jeremy Irvin Lucas Tao Joanne Zhou Yuntao Ma Langston Nashold Benjamin Liu Andrew Y. Ng ViT 30 19 0 02 Dec 2023
End-to-End Temporal Action Detection with 1B Parameters Across 1000 Frames Shuming Liu Chen-Da Liu-Zhang Chen Zhao Bernard Ghanem 24 25 0 28 Nov 2023
LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models Yanwei Li Chengyao Wang Jiaya Jia VLM MLLM 26 259 0 28 Nov 2023
Optimizing and Fine-tuning Large Language Model for Urban Renewal Xi Wang Xianyao Ling Tom Zhang Xuecao Li Shaolan Wang Zhixing Li Liang Zhang Peng Gong 63 7 0 27 Nov 2023
Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey Yunpeng Huang Jingwei Xu Junyu Lai Zixu Jiang Taolue Chen ... Xiaoxing Ma Lijuan Yang Zhou Xin Shupeng Li Penghao Zhao LLMAG KELM 28 53 0 21 Nov 2023
AcademicGPT: Empowering Academic Research Shufa Wei Xiaolong Xu Xianbiao Qi Xi Yin Jun Xia ... Chihao Dai Lihua Wang Xiaohui Liu Lei Zhang Yutao Xie LM&MA 39 3 0 21 Nov 2023
Long-MIL: Scaling Long Contextual Multiple Instance Learning for Histopathology Whole Slide Image Analysis Honglin Li Yunlong Zhang Chenglu Zhu Jiatong Cai Sunyi Zheng Lin Yang VLM 25 4 0 21 Nov 2023
LongQLoRA: Efficient and Effective Method to Extend Context Length of Large Language Models Jianxin Yang 17 6 0 08 Nov 2023
LooGLE: Can Long-Context Language Models Understand Long Contexts? Jiaqi Li Mengmeng Wang Zilong Zheng Muhan Zhang ELM RALM 24 106 0 08 Nov 2023
Navigating Scaling Laws: Compute Optimality in Adaptive Model Training Sotiris Anagnostidis Gregor Bachmann Imanol Schlag Thomas Hofmann 23 2 0 06 Nov 2023
PACuna: Automated Fine-Tuning of Language Models for Particle Accelerators Antonin Sulc Raimund Kammering Annika Eichler T. Wilksen 21 3 0 29 Oct 2023
End-to-end Story Plot Generator Hanlin Zhu Andrew Cohen Danqing Wang Kevin Kaichuang Yang Xiaomeng Yang Jiantao Jiao Yuandong Tian 22 5 0 13 Oct 2023
PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training Dawei Zhu Nan Yang Liang Wang Yifan Song Wenhao Wu Furu Wei Sujian Li 55 77 0 19 Sep 2023
A Comprehensive Overview of Large Language Models Humza Naveed Asad Ullah Khan Shi Qiu Muhammad Saqib Saeed Anwar Muhammad Usman Naveed Akhtar Nick Barnes Ajmal Saeed Mian OffRL 46 499 0 12 Jul 2023
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press Noah A. Smith M. Lewis 242 690 0 27 Aug 2021
The Power of Scale for Parameter-Efficient Prompt Tuning Brian Lester Rami Al-Rfou Noah Constant VPVLM 278 3,784 0 18 Apr 2021
Big Bird: Transformers for Longer Sequences Manzil Zaheer Guru Guruganesh Kumar Avinava Dubey Joshua Ainslie Chris Alberti ... Philip Pham Anirudh Ravula Qifan Wang Li Yang Amr Ahmed VLM 249 1,982 0 28 Jul 2020