Simple and Scalable Strategies to Continually Pre-train Large Language Models

13 March 2024

Papers citing "Simple and Scalable Strategies to Continually Pre-train Large Language Models"

22 / 22 papers shown

Title
Elastic Weight Consolidation for Full-Parameter Continual Pre-Training of Gemma2 Vytenis Šliogeris Povilas Daniušis Arturas Nakvosas CLL 21 0 0 09 May 2025
Bielik 11B v2 Technical Report Krzysztof Ociepa Łukasz Flis Krzysztof Wróbel Adrian Gwoździej Remigiusz Kinas 22 0 0 05 May 2025
Bielik v3 Small: Technical Report Krzysztof Ociepa Łukasz Flis Remigiusz Kinas Krzysztof Wróbel Adrian Gwoździej 15 0 0 05 May 2025
WenyanGPT: A Large Language Model for Classical Chinese Tasks Xinyu Yao Mengdi Wang Bo Chen Xiaobing Zhao 60 0 0 29 Apr 2025
Llama-3.1-FoundationAI-SecurityLLM-Base-8B Technical Report Paul Kassianik Baturay Saglam Alexander Chen Blaine Nelson Anu Vellore ... Hyrum Anderson Kojin Oshiba Omar Santos Yaron Singer Amin Karbasi PILM 51 0 0 28 Apr 2025
Task-conditioned Ensemble of Expert Models for Continuous Learning Renu Sharma Debasmita Pal Arun Ross OOD KELM 50 0 0 11 Apr 2025
Time Transfer: On Optimal Learning Rate and Batch Size In The Infinite Data Limit Oleg Filatov Jan Ebert Jiangtao Wang Stefan Kesselheim 26 3 0 10 Jan 2025
Yi-Lightning Technical Report 01. AI : Alan Wake Albert Wang Bei Chen ... Yuxuan Sha Zhaodong Yan Zhiyuan Liu Zirui Zhang Zonghong Dai OSLM 84 3 0 02 Dec 2024
MIND: Math Informed syNthetic Dialogues for Pretraining LLMs Syeda Nahida Akter Shrimai Prabhumoye John Kamalu S. Satheesh Eric Nyberg M. Patwary M. Shoeybi Bryan Catanzaro LRM SyDa ReLM 78 1 0 15 Oct 2024
CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning Yibo Yang Xiaojie Li Zhongzhu Zhou S. Song Jianlong Wu Liqiang Nie Bernard Ghanem 36 6 0 07 Jun 2024
Building a Large Japanese Web Corpus for Large Language Models Naoaki Okazaki Kakeru Hattori Hirai Shota Hiroki Iida Masanari Ohi Kazuki Fujii Taishi Nakamura Mengsay Loem Rio Yokota Sakae Mizuki 29 6 0 27 Apr 2024
CtRL-Sim: Reactive and Controllable Driving Agents with Offline Reinforcement Learning Luke Rowe Roger Girgis Anthony Gosselin Bruno Carrez Florian Golemo Felix Heide Liam Paull Christopher Pal 18 4 0 29 Mar 2024
The Falcon Series of Open Language Models Ebtesam Almazrouei Hamza Alobeidli Abdulaziz Alshamsi Alessandro Cappelli Ruxandra-Aimée Cojocaru ... Quentin Malartic Daniele Mazzotta Badreddine Noune B. Pannier Guilherme Penedo AI4TS ALM 104 389 0 28 Nov 2023
Efficient Continual Pre-training for Building Domain Specific Large Language Models Yong Xie Karan Aggarwal Aitzaz Ahmad CLL 29 7 0 14 Nov 2023
DiLoCo: Distributed Low-Communication Training of Language Models Arthur Douillard Qixuang Feng Andrei A. Rusu Rachita Chhaparia Yani Donchev A. Kuncoro MarcÁurelio Ranzato Arthur Szlam Jiajun Shen 38 12 0 14 Nov 2023
Online Continual Learning Without the Storage Constraint Ameya Prabhu Zhipeng Cai P. Dokania Philip H. S. Torr V. Koltun Ozan Sener CLL 107 28 0 16 May 2023
Continual evaluation for lifelong learning: Identifying the stability gap Matthias De Lange Gido M. van de Ven Tinne Tuytelaars CLL 77 28 0 26 May 2022
Fine-tuned Language Models are Continual Learners Thomas Scialom Tuhin Chakrabarty Smaranda Muresan CLL LRM 132 116 0 24 May 2022
Towards Continual Knowledge Learning of Language Models Joel Jang Seonghyeon Ye Sohee Yang Joongbo Shin Janghoon Han Gyeonghun Kim Stanley Jungkyu Choi Minjoon Seo CLL KELM 216 122 0 07 Oct 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 236 1,508 0 31 Dec 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 220 3,054 0 23 Jan 2020
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 243 1,791 0 17 Sep 2019