Large Batch Optimization for Deep Learning: Training BERT in 76 minutes

1 April 2019

Yang You

Jing Li

Sashank J. Reddi

Jonathan Hseu

Sanjiv Kumar

Srinadh Bhojanapalli

Papers citing "Large Batch Optimization for Deep Learning: Training BERT in 76 minutes"

50 / 170 papers shown

Title
mSAM: Micro-Batch-Averaged Sharpness-Aware Minimization Kayhan Behdin Qingquan Song Aman Gupta S. Keerthi Ayan Acharya Borja Ocejo Gregory Dexter Rajiv Khanna D. Durfee Rahul Mazumder AAML 13 7 0 19 Feb 2023
Improving Training Stability for Multitask Ranking Models in Recommender Systems Jiaxi Tang Yoel Drori Daryl Chang M. Sathiamoorthy Justin Gilmer Li Wei Xinyang Yi Lichan Hong Ed H. Chi 17 10 0 17 Feb 2023
DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedule Maor Ivgi Oliver Hinder Y. Carmon ODL 22 56 0 08 Feb 2023
A Survey on Efficient Training of Transformers Bohan Zhuang Jing Liu Zizheng Pan Haoyu He Yuetian Weng Chunhua Shen 18 47 0 02 Feb 2023
A Survey of Deep Learning: From Activations to Transformers Johannes Schneider Michalis Vlachos ViT MedIm AI4TS AI4CE 38 9 0 01 Feb 2023
STAIR: Learning Sparse Text and Image Representation in Grounded Tokens Chen Chen Bowen Zhang Liangliang Cao Jiguang Shen Tom Gunter Albin Madappally Jose Alexander Toshev Jonathon Shlens Ruoming Pang Yinfei Yang VLM 3DV 17 13 0 30 Jan 2023
SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient Max Ryabinin Tim Dettmers Michael Diskin Alexander Borzunov MoE 22 31 0 27 Jan 2023
Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling Keyu Tian Yi-Xin Jiang Qishuai Diao Chen Lin Liwei Wang Zehuan Yuan 25 100 0 09 Jan 2023
Sharper Analysis for Minibatch Stochastic Proximal Point Methods: Stability, Smoothness, and Deviation Xiao-Tong Yuan P. Li 22 2 0 09 Jan 2023
Bort: Towards Explainable Neural Networks with Bounded Orthogonal Constraint Borui Zhang Wenzhao Zheng Jie Zhou Jiwen Lu AAML 23 7 0 18 Dec 2022
Deep Incubation: Training Large Models by Divide-and-Conquering Zanlin Ni Yulin Wang Jiangwei Yu Haojun Jiang Yu Cao Gao Huang VLM 18 11 0 08 Dec 2022
PipeFisher: Efficient Training of Large Language Models Using Pipelining and Fisher Information Matrices Kazuki Osawa Shigang Li Torsten Hoefler AI4CE 33 24 0 25 Nov 2022
Differentially Private Image Classification from Features Harsh Mehta Walid Krichene Abhradeep Thakurta Alexey Kurakin Ashok Cutkosky 38 7 0 24 Nov 2022
Q-Ensemble for Offline RL: Don't Scale the Ensemble, Scale the Batch Size Alexander Nikulin Vladislav Kurenkov Denis Tarasov Dmitry Akimov Sergey Kolesnikov OffRL 12 14 0 20 Nov 2022
SeDR: Segment Representation Learning for Long Documents Dense Retrieval Junying Chen Qingcai Chen Dongfang Li Yutao Huang 14 6 0 20 Nov 2022
VeLO: Training Versatile Learned Optimizers by Scaling Up Luke Metz James Harrison C. Freeman Amil Merchant Lucas Beyer ... Naman Agrawal Ben Poole Igor Mordatch Adam Roberts Jascha Narain Sohl-Dickstein 21 60 0 17 Nov 2022
Mask More and Mask Later: Efficient Pre-training of Masked Language Models by Disentangling the [MASK] Token Baohao Liao David Thulke Sanjika Hewavitharana Hermann Ney Christof Monz 28 9 0 09 Nov 2022
Conciseness: An Overlooked Language Task Felix Stahlberg Aashish Kumar Chris Alberti Shankar Kumar 16 1 0 08 Nov 2022
Adaptive Compression for Communication-Efficient Distributed Training Maksim Makarenko Elnur Gasanov Rustem Islamov Abdurakhmon Sadiev Peter Richtárik 18 12 0 31 Oct 2022
MetaFormer Baselines for Vision Weihao Yu Chenyang Si Pan Zhou Mi Luo Yichen Zhou Jiashi Feng Shuicheng Yan Xinchao Wang MoE 23 156 0 24 Oct 2022
lo-fi: distributed fine-tuning without communication Mitchell Wortsman Suchin Gururangan Shen Li Ali Farhadi Ludwig Schmidt Michael G. Rabbat Ari S. Morcos 19 24 0 19 Oct 2022
Learning image representations for anomaly detection: application to discovery of histological alterations in drug development I. Zingman B. Stierstorfer C. Lempp Fabian Heinemann OOD MedIm 17 11 0 14 Oct 2022
VER: Scaling On-Policy RL Leads to the Emergence of Navigation in Embodied Rearrangement Erik Wijmans Irfan Essa Dhruv Batra OffRL 28 13 0 11 Oct 2022
CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-training Tianyu Huang Bowen Dong Yunhan Yang Xiaoshui Huang Rynson W. H. Lau Wanli Ouyang W. Zuo VLM 3DPC CLIP 36 144 0 03 Oct 2022
A patch-based architecture for multi-label classification from single label annotations Warren Jouanneau Aurélie Bugeau Marc Palyart Nicolas Papadakis Laurent Vézard 28 0 0 14 Sep 2022
Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation Mohit Shridhar Lucas Manuelli D. Fox LM&Ro 155 456 0 12 Sep 2022
On the Factory Floor: ML Engineering for Industrial-Scale Ads Recommendation Models Rohan Anil S. Gadanho Danya Huang Nijith Jacob Zhuoshu Li ... Cristina Pop Kevin Regan G. Shamir Rakesh Shivanna Qiqi Yan 3DV 8 41 0 12 Sep 2022
LATTE: LAnguage Trajectory TransformEr A. Bucker Luis F. C. Figueredo Sami Haddadin Ashish Kapoor Shuang Ma Sai H. Vemprala Rogerio Bonatti LM&Ro 22 59 0 04 Aug 2022
CloudAttention: Efficient Multi-Scale Attention Scheme For 3D Point Cloud Learning Mahdi Saleh Yige Wang Nassir Navab Benjamin Busam F. Tombari 3DPC 21 3 0 31 Jul 2022
Scalable K-FAC Training for Deep Neural Networks with Distributed Preconditioning Lin Zhang S. Shi Wei Wang Bo-wen Li 28 10 0 30 Jun 2022
RetrieverTTS: Modeling Decomposed Factors for Text-Based Speech Insertion Dacheng Yin Chuanxin Tang Yanqing Liu Xiaoqiang Wang Zhiyuan Zhao Yucheng Zhao Zhiwei Xiong Sheng Zhao Chong Luo 16 12 0 28 Jun 2022
Deep Learning Models on CPUs: A Methodology for Efficient Training Quchen Fu Ramesh Chukka Keith Achorn Thomas Atta-fosu Deepak R. Canchi Zhongwei Teng Jules White Douglas C. Schmidt 16 1 0 20 Jun 2022
Distributed Adversarial Training to Robustify Deep Neural Networks at Scale Gaoyuan Zhang Songtao Lu Yihua Zhang Xiangyi Chen Pin-Yu Chen Quanfu Fan Lee Martie L. Horesh Min-Fong Hong Sijia Liu OOD 22 12 0 13 Jun 2022
Dataset Distillation using Neural Feature Regression Yongchao Zhou E. Nezhadarya Jimmy Ba DD FedML 35 149 0 01 Jun 2022
Efficient-Adam: Communication-Efficient Distributed Adam Congliang Chen Li Shen Wei Liu Z. Luo 23 19 0 28 May 2022
Architecture-Agnostic Masked Image Modeling -- From ViT back to CNN Siyuan Li Di Wu Fang Wu Lei Shang Stan.Z.Li 32 47 0 27 May 2022
TransBoost: Improving the Best ImageNet Performance using Deep Transduction Omer Belhasin Guy Bar-Shalom Ran El-Yaniv ViT 19 3 0 26 May 2022
Amortized Inference for Causal Structure Learning Lars Lorch Scott Sussex Jonas Rothfuss Andreas Krause Bernhard Schölkopf CML 11 60 0 25 May 2022
Life after BERT: What do Other Muppets Understand about Language? Vladislav Lialin Kevin Zhao Namrata Shivagunde Anna Rumshisky 34 6 0 21 May 2022
Large Neural Networks Learning from Scratch with Very Few Data and without Explicit Regularization C. Linse T. Martinetz SSL VLM 10 4 0 18 May 2022
CowClip: Reducing CTR Prediction Model Training Time from 12 hours to 10 minutes on 1 GPU Zangwei Zheng Peng Xu Xuan Zou Da Tang Zhen Li ... Xiangzhuo Ding Fuzhao Xue Ziheng Qing Youlong Cheng Yang You VLM 37 7 0 13 Apr 2022
PICASSO: Unleashing the Potential of GPU-centric Training for Wide-and-deep Recommender Systems Yuanxing Zhang Langshi Chen Siran Yang Man Yuan Hui-juan Yi ... Yong Li Dingyang Zhang Wei Lin Lin Qu Bo Zheng 22 32 0 11 Apr 2022
Solving ImageNet: a Unified Scheme for Training any Backbone to Top Results T. Ridnik Hussam Lawen Emanuel Ben-Baruch Asaf Noy 31 11 0 07 Apr 2022
Reshaping Robot Trajectories Using Natural Language Commands: A Study of Multi-Modal Data Alignment Using Transformers A. Bucker Luis F. C. Figueredo Sami Haddadin Ashish Kapoor Shuang Ma Rogerio Bonatti LM&Ro 14 49 0 25 Mar 2022
A DNN Optimizer that Improves over AdaBelief by Suppression of the Adaptive Stepsize Range Guoqiang Zhang Kenta Niwa W. Kleijn ODL 11 2 0 24 Mar 2022
Practical tradeoffs between memory, compute, and performance in learned optimizers Luke Metz C. Freeman James Harrison Niru Maheswaranathan Jascha Narain Sohl-Dickstein 31 32 0 22 Mar 2022
Harnessing Hard Mixed Samples with Decoupled Regularizer Zicheng Liu Siyuan Li Ge Wang Cheng Tan Lirong Wu Stan Z. Li 51 18 0 21 Mar 2022
CYBORGS: Contrastively Bootstrapping Object Representations by Grounding in Segmentation Renhao Wang Hang Zhao Yang Gao SSL 14 1 0 17 Mar 2022
Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifiers Guodong Zhang Aleksandar Botev James Martens OffRL 15 26 0 15 Mar 2022
ELLE: Efficient Lifelong Pre-training for Emerging Data Yujia Qin Jiajie Zhang Yankai Lin Zhiyuan Liu Peng Li Maosong Sun Jie Zhou 14 67 0 12 Mar 2022