Continual Pre-training of Language Models

7 February 2023

Papers citing "Continual Pre-training of Language Models"

26 / 26 papers shown

Title
Synthesize-on-Graph: Knowledgeable Synthetic Data Generation for Continue Pre-training of Large Language Models Xuhui Jiang Shengjie Ma Chengjin Xu Cehao Yang Liyu Zhang Jian Guo SyDa 28 0 0 02 May 2025
EnronQA: Towards Personalized RAG over Private Documents Michael J. Ryan Danmei Xu Chris Nivera Daniel Campos SILM 62 0 0 01 May 2025
WenyanGPT: A Large Language Model for Classical Chinese Tasks Xinyu Yao Mengdi Wang Bo Chen Xiaobing Zhao 67 0 0 29 Apr 2025
Efficient Domain-adaptive Continual Pretraining for the Process Industry in the German Language Anastasia Zhukova Christian E. Matt Terry Ruas Bela Gipp CLL VLM 98 0 0 28 Apr 2025
Param $Δ$ for Direct Weight Mixing: Post-Train Large Language Model at Zero Cost Sheng Cao Mingrui Wu Karthik Prasad Yuandong Tian Zechun Liu MoMe 74 0 0 23 Apr 2025
ProtTeX: Structure-In-Context Reasoning and Editing of Proteins with Large Language Models Zicheng Ma Chuanliu Fan Zhicong Wang Zhenyu Chen Xiaohan Lin Y. Li Shihao Feng Jun Zhang Ziqiang Cao Y. Gao 43 0 0 11 Mar 2025
Exploring Graph Tasks with Pure LLMs: A Comprehensive Benchmark and Investigation Y. Wang Xinnan Dai Wenqi Fan Yao Ma 67 1 0 26 Feb 2025
Unlocking the Power of Function Vectors for Characterizing and Mitigating Catastrophic Forgetting in Continual Instruction Tuning Gangwei Jiang Caigao Jiang Zhaoyi Li Siqiao Xue Jun-ping Zhou Linqi Song Defu Lian Yin Wei CLL MU 56 0 0 16 Feb 2025
Scaling Diffusion Language Models via Adaptation from Autoregressive Models Shansan Gong Shivam Agarwal Yizhe Zhang Jiacheng Ye Lin Zheng ... Peilin Zhao W. Bi Jiawei Han Hao Peng Lingpeng Kong AI4CE 70 14 0 23 Oct 2024
From Tokens to Words: On the Inner Lexicon of LLMs Guy Kaplan Matanel Oren Yuval Reif Roy Schwartz 41 12 0 08 Oct 2024
Towards LifeSpan Cognitive Systems Yu Wang Chi Han Tongtong Wu Xiaoxin He Wangchunshu Zhou ... Zexue He Wei Wang Gholamreza Haffari Heng Ji Julian McAuley KELM CLL 88 1 0 20 Sep 2024
MindGuard: Towards Accessible and Sitgma-free Mental Health First Aid via Edge LLM Sijie Ji Xinzhe Zheng Jiawei Sun Renqi Chen Wei Gao Mani Srivastava AI4MH 32 2 0 16 Sep 2024
An Investigation of Warning Erroneous Chat Translations in Cross-lingual Communication Yunmeng Li Jun Suzuki Makoto Morishita Kaori Abe Kentaro Inui 53 1 0 28 Aug 2024
Extend Model Merging from Fine-Tuned to Pre-Trained Large Language Models via Weight Disentanglement Le Yu Bowen Yu Haiyang Yu Fei Huang Yongbin Li MoMe 27 5 0 06 Aug 2024
MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuning Yupeng Chen Senmiao Wang Zhihang Lin Zhihang Lin Yushun Zhang Tian Ding Ruoyu Sun Ruoyu Sun CLL 72 1 0 30 Jul 2024
A Survey on Large Language Models from General Purpose to Medical Applications: Datasets, Methodologies, and Evaluations Jinqiang Wang Huansheng Ning Yi Peng Qikai Wei Daniel Tesfai Wenwei Mao Tao Zhu Runhe Huang LM&MA AI4MH ELM 36 4 0 14 Jun 2024
From Matching to Generation: A Survey on Generative Information Retrieval Xiaoxi Li Jiajie Jin Yujia Zhou Yuyao Zhang Peitian Zhang Yutao Zhu Zhicheng Dou 3DV 67 45 0 23 Apr 2024
Hyperparameters in Continual Learning: A Reality Check Sungmin Cha Kyunghyun Cho CLL 68 2 0 14 Mar 2024
Investigating Continual Pretraining in Large Language Models: Insights and Implications cCaugatay Yildiz Nishaanth Kanna Ravichandran Prishruit Punia Matthias Bethge B. Ermiş CLL KELM LRM 46 25 0 27 Feb 2024
Continual Pre-Training of Large Language Models: How to (re)warm your model? Kshitij Gupta Benjamin Thérien Adam Ibrahim Mats L. Richter Quentin G. Anthony Eugene Belilovsky Irina Rish Timothée Lesort KELM 22 98 0 08 Aug 2023
Continual Training of Language Models for Few-Shot Learning Zixuan Ke Haowei Lin Yijia Shao Hu Xu Lei Shu Bin Liu KELM BDL CLL 85 33 0 11 Oct 2022
Adapting BERT for Continual Learning of a Sequence of Aspect Sentiment Classification Tasks Zixuan Ke Hu Xu Bing-Quan Liu CLL 222 81 0 06 Dec 2021
Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning Zixuan Ke Bing-Quan Liu Nianzu Ma Hu Xu Lei Shu CLL 178 121 0 05 Dec 2021
Fast Model Editing at Scale E. Mitchell Charles Lin Antoine Bosselut Chelsea Finn Christopher D. Manning KELM 219 341 0 21 Oct 2021
Towards Continual Knowledge Learning of Language Models Joel Jang Seonghyeon Ye Sohee Yang Joongbo Shin Janghoon Han Gyeonghun Kim Stanley Jungkyu Choi Minjoon Seo CLL KELM 222 150 0 07 Oct 2021
The Power of Scale for Parameter-Efficient Prompt Tuning Brian Lester Rami Al-Rfou Noah Constant VPVLM 278 3,835 0 18 Apr 2021