Norm matters: efficient and accurate normalization schemes in deep networks

5 March 2018

Papers citing "Norm matters: efficient and accurate normalization schemes in deep networks"

31 / 31 papers shown

Title
Normalization and effective learning rates in reinforcement learning Clare Lyle Zeyu Zheng Khimya Khetarpal James Martens H. V. Hasselt Razvan Pascanu Will Dabney 19 7 0 01 Jul 2024
$Implicit Bias of AdamW: $\ell_\infty$ Norm Constrained Optimization$ Implicit Bias of AdamW: $\ell_\infty$ Norm Constrained Optimization Shuo Xie Zhiyuan Li OffRL 44 12 0 05 Apr 2024
Analyzing and Improving the Training Dynamics of Diffusion Models Tero Karras M. Aittala J. Lehtinen Janne Hellsten Timo Aila S. Laine 28 155 0 05 Dec 2023
The Implicit Bias of Batch Normalization in Linear Models and Two-layer Linear Convolutional Neural Networks Yuan Cao Difan Zou Yuan-Fang Li Quanquan Gu MLT 29 5 0 20 Jun 2023
On the Weight Dynamics of Deep Normalized Networks Christian H. X. Ali Mehmeti-Göpel Michael Wand 30 1 0 01 Jun 2023
Exploiting the Partly Scratch-off Lottery Ticket for Quantization-Aware Training Yunshan Zhong Gongrui Nan Yu-xin Zhang Fei Chao Rongrong Ji MQ 18 3 0 12 Nov 2022
Toward Equation of Motion for Deep Neural Networks: Continuous-time Gradient Descent and Discretization Error Analysis Taiki Miyagawa 50 9 0 28 Oct 2022
Noise Injection as a Probe of Deep Learning Dynamics Noam Levi I. Bloch M. Freytsis T. Volansky 37 2 0 24 Oct 2022
Adapting the Linearised Laplace Model Evidence for Modern Deep Learning Javier Antorán David Janz J. Allingham Erik A. Daxberger Riccardo Barbano Eric T. Nalisnick José Miguel Hernández-Lobato UQCV BDL 27 28 0 17 Jun 2022
Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction Kaifeng Lyu Zhiyuan Li Sanjeev Arora FAtt 37 69 0 14 Jun 2022
Compression-aware Training of Neural Networks using Frank-Wolfe Max Zimmer Christoph Spiegel S. Pokutta 21 9 0 24 May 2022
Guidelines for the Regularization of Gammas in Batch Normalization for Deep Residual Networks Bum Jun Kim Hyeyeon Choi Hyeonah Jang Dong Gu Lee Wonseok Jeong Sang Woo Kim 16 4 0 15 May 2022
Robust Training of Neural Networks Using Scale Invariant Architectures Zhiyuan Li Srinadh Bhojanapalli Manzil Zaheer Sashank J. Reddi Surinder Kumar 19 27 0 02 Feb 2022
Logit Attenuating Weight Normalization Aman Gupta R. Ramanath Jun Shi Anika Ramachandran Sirou Zhou Mingzhou Zhou S. Keerthi 34 1 0 12 Aug 2021
Large-Scale Differentially Private BERT Rohan Anil Badih Ghazi Vineet Gupta Ravi Kumar Pasin Manurangsi 36 131 0 03 Aug 2021
Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspective Florin Gogianu Tudor Berariu Mihaela Rosca Claudia Clopath L. Buşoniu Razvan Pascanu 18 52 0 11 May 2021
Initialization and Regularization of Factorized Neural Layers M. Khodak Neil A. Tenenholtz Lester W. Mackey Nicolò Fusi 65 56 0 03 May 2021
Enabling Binary Neural Network Training on the Edge Erwei Wang James J. Davis Daniele Moro Piotr Zielinski Jia Jie Lim C. Coelho S. Chatterjee P. Cheung G. Constantinides MQ 20 24 0 08 Feb 2021
Advances in Electron Microscopy with Deep Learning Jeffrey M. Ede 32 2 0 04 Jan 2021
Dissipative Deep Neural Dynamical Systems Ján Drgoňa Soumya Vasisht Aaron Tuor D. Vrabie 19 6 0 26 Nov 2020
Review: Deep Learning in Electron Microscopy Jeffrey M. Ede 31 79 0 17 Sep 2020
GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training Tianle Cai Shengjie Luo Keyulu Xu Di He Tie-Yan Liu Liwei Wang GNN 23 158 0 07 Sep 2020
Quantum State Tomography with Conditional Generative Adversarial Networks Shahnawaz Ahmed C. Muñoz Franco Nori A. F. Kockum GAN 3DPC 45 121 0 07 Aug 2020
New Interpretations of Normalization Methods in Deep Learning Jiacheng Sun Xiangyong Cao Hanwen Liang Weiran Huang Zewei Chen Zhenguo Li 21 34 0 16 Jun 2020
Shape Matters: Understanding the Implicit Bias of the Noise Covariance Jeff Z. HaoChen Colin Wei J. Lee Tengyu Ma 29 93 0 15 Jun 2020
On the training dynamics of deep networks with $L_2$ regularization Aitor Lewkowycz Guy Gur-Ari 36 53 0 15 Jun 2020
Evolving Normalization-Activation Layers Hanxiao Liu Andrew Brock Karen Simonyan Quoc V. Le 12 79 0 06 Apr 2020
Iterative Averaging in the Quest for Best Test Error Diego Granziol Xingchen Wan Samuel Albanie Stephen J. Roberts 10 3 0 02 Mar 2020
Switchable Normalization for Learning-to-Normalize Deep Representation Ping Luo Ruimao Zhang Jiamin Ren Zhanglin Peng Jingyu Li 30 73 0 22 Jul 2019
Theoretical Analysis of Auto Rate-Tuning by Batch Normalization Sanjeev Arora Zhiyuan Li Kaifeng Lyu 26 130 0 10 Dec 2018
Scalable Methods for 8-bit Training of Neural Networks Ron Banner Itay Hubara Elad Hoffer Daniel Soudry MQ 37 331 0 25 May 2018