Worldwide Federated Training of Language Models

Worldwide Federated Training of Language Models

23 May 2024

Alexandru Iacob

Preslav Aleksandrov

William F. Shen

Nicholas D. Lane

Papers citing "Worldwide Federated Training of Language Models"

9 / 9 papers shown

Title
NoEsis: Differentially Private Knowledge Transfer in Modular LLM Adaptation Rob Romijnders Stefanos Laskaridis Ali Shahin Shamsabadi Hamed Haddadi 54 0 0 25 Apr 2025
DiLoCo: Distributed Low-Communication Training of Language Models Arthur Douillard Qixuang Feng Andrei A. Rusu Rachita Chhaparia Yani Donchev A. Kuncoro MarcÁurelio Ranzato Arthur Szlam Jiajun Shen 49 12 0 14 Nov 2023
Dataset Geography: Mapping Language Data to Language Users Fahim Faisal Yinkai Wang Antonios Anastasopoulos 48 23 0 07 Dec 2021
Papaya: Practical, Private, and Scalable Federated Learning Dzmitry Huba John Nguyen Kshitiz Malik Ruiyu Zhu Michael G. Rabbat ... H. Srinivas Kaikai Wang Anthony Shoumikhin Jesik Min Mani Malek FedML 97 133 0 08 Nov 2021
LightSecAgg: a Lightweight and Versatile Design for Secure Aggregation in Federated Learning Jinhyun So Chaoyang He Chien-Sheng Yang Songze Li Qian-long Yu Ramy E. Ali Başak Güler Salman Avestimehr FedML 55 163 0 29 Sep 2021
Federated Learning on Non-IID Data Silos: An Experimental Study Q. Li Yiqun Diao Quan Chen Bingsheng He FedML OOD 82 936 0 03 Feb 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 239 1,508 0 31 Dec 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 220 3,054 0 23 Jan 2020
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima N. Keskar Dheevatsa Mudigere J. Nocedal M. Smelyanskiy P. T. P. Tang ODL 273 2,696 0 15 Sep 2016