Lifelong Pretraining: Continually Adapting Language Models to Emerging
Corpora

Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora

16 October 2021

Andrew O. Arnold

Xiang Ren

Papers citing "Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora"

19 / 19 papers shown

Title
TiC-LM: A Web-Scale Benchmark for Time-Continual LLM Pretraining Jeffrey Li Mohammadreza Armandpour Iman Mirzadeh Sachin Mehta Vaishaal Shankar ... Samy Bengio Oncel Tuzel Mehrdad Farajtabar Hadi Pouransari Fartash Faghri CLL KELM 59 0 0 02 Apr 2025
Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Models Yulei Qin Yuncheng Yang Pengcheng Guo Gang Li Hang Shao Yuchen Shi Zihan Xu Yun Gu Ke Li Xing Sun ALM 88 12 0 31 Dec 2024
Gradient Localization Improves Lifelong Pretraining of Language Models Jared Fernandez Yonatan Bisk Emma Strubell KELM 31 1 0 07 Nov 2024
An Investigation of Warning Erroneous Chat Translations in Cross-lingual Communication Yunmeng Li Jun Suzuki Makoto Morishita Kaori Abe Kentaro Inui 53 1 0 28 Aug 2024
Language Modeling with Editable External Knowledge Belinda Z. Li Emmy Liu Alexis Ross Abbas Zeitoun Graham Neubig Jacob Andreas KELM 25 4 0 17 Jun 2024
Investigating Continual Pretraining in Large Language Models: Insights and Implications cCaugatay Yildiz Nishaanth Kanna Ravichandran Prishruit Punia Matthias Bethge B. Ermiş CLL KELM LRM 46 25 0 27 Feb 2024
Online Continual Knowledge Learning for Language Models Yuhao Wu Tongjun Shi Karthick Sharma Chun Seah Shuhao Zhang CLL KELM 21 4 0 16 Nov 2023
Goodtriever: Adaptive Toxicity Mitigation with Retrieval-augmented Models Luiza Amador Pozzobon B. Ermiş Patrick Lewis Sara Hooker 28 20 0 11 Oct 2023
Continual Pre-Training of Large Language Models: How to (re)warm your model? Kshitij Gupta Benjamin Thérien Adam Ibrahim Mats L. Richter Quentin G. Anthony Eugene Belilovsky Irina Rish Timothée Lesort KELM 22 99 0 08 Aug 2023
PreCog: Exploring the Relation between Memorization and Performance in Pre-trained Language Models Leonardo Ranaldi Elena Sofia Ruzzetti Fabio Massimo Zanzotto 23 6 0 08 May 2023
An Overview on Language Models: Recent Developments and Outlook Chengwei Wei Yun Cheng Wang Bin Wang C.-C. Jay Kuo 15 41 0 10 Mar 2023
Preventing Catastrophic Forgetting in Continual Learning of New Natural Language Tasks Sudipta Kar Giuseppe Castellucci Simone Filice S. Malmasi Oleg Rokhlenko CLL KELM 46 6 0 22 Feb 2023
He Said, She Said: Style Transfer for Shifting the Perspective of Dialogues Amanda Bertsch Graham Neubig Matthew R. Gormley 40 5 0 27 Oct 2022
TempoWiC: An Evaluation Benchmark for Detecting Meaning Shift in Social Media Daniel Loureiro Aminette D'Souza Areej Muhajab Isabella A. White Gabriel Wong Luis Espinosa Anke Leonardo Neves Francesco Barbieri Jose Camacho-Collados 27 25 0 15 Sep 2022
Fine-tuned Language Models are Continual Learners Thomas Scialom Tuhin Chakrabarty Smaranda Muresan CLL LRM 139 116 0 24 May 2022
Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models Ning Ding Yujia Qin Guang Yang Fu Wei Zonghan Yang ... Jianfei Chen Yang Liu Jie Tang Juan Li Maosong Sun 13 196 0 14 Mar 2022
TimeLMs: Diachronic Language Models from Twitter Daniel Loureiro Francesco Barbieri Leonardo Neves Luis Espinosa Anke Jose Camacho-Collados 10 247 0 08 Feb 2022
Towards Continual Knowledge Learning of Language Models Joel Jang Seonghyeon Ye Sohee Yang Joongbo Shin Janghoon Han Gyeonghun Kim Stanley Jungkyu Choi Minjoon Seo CLL KELM 222 150 0 07 Oct 2021
SEED: Self-supervised Distillation For Visual Representation Zhiyuan Fang Jianfeng Wang Lijuan Wang Lei Zhang Yezhou Yang Zicheng Liu SSL 231 190 0 12 Jan 2021