A Closer Look at Codistillation for Distributed Training

6 October 2020

Papers citing "A Closer Look at Codistillation for Distributed Training"

8 / 8 papers shown

Title
Scalable Collaborative Learning via Representation Sharing Frédéric Berdoz Abhishek Singh Martin Jaggi Ramesh Raskar FedML 17 3 0 20 Nov 2022
lo-fi: distributed fine-tuning without communication Mitchell Wortsman Suchin Gururangan Shen Li Ali Farhadi Ludwig Schmidt Michael G. Rabbat Ari S. Morcos 19 24 0 19 Oct 2022
Towards Model Agnostic Federated Learning Using Knowledge Distillation A. Afonin Sai Praneeth Karimireddy FedML 30 44 0 28 Oct 2021
Personalized Federated Learning for Heterogeneous Clients with Clustered Knowledge Transfer Yae Jee Cho Jianyu Wang Tarun Chiruvolu Gauri Joshi FedML 27 29 0 16 Sep 2021
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 226 4,453 0 23 Jan 2020
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 243 1,817 0 17 Sep 2019
Bag of Tricks for Image Classification with Convolutional Neural Networks Tong He Zhi-Li Zhang Hang Zhang Zhongyue Zhang Junyuan Xie Mu Li 216 1,398 0 04 Dec 2018
Large scale distributed neural network training through online distillation Rohan Anil Gabriel Pereyra Alexandre Passos Róbert Ormándi George E. Dahl Geoffrey E. Hinton FedML 267 404 0 09 Apr 2018