BranchNorm: Robustly Scaling Extremely Deep Transformers

4 May 2023

Jie Zhou

Papers citing "BranchNorm: Robustly Scaling Extremely Deep Transformers"

3 / 3 papers shown

Title
M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining Junyang Lin An Yang Jinze Bai Chang Zhou Le Jiang ... Jie M. Zhang Yong Li Wei Lin Jingren Zhou Hongxia Yang MoE 84 43 0 08 Oct 2021
Larger-Scale Transformers for Multilingual Masked Language Modeling Naman Goyal Jingfei Du Myle Ott Giridhar Anantharaman Alexis Conneau 88 126 0 02 May 2021
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 226 4,424 0 23 Jan 2020