Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks

4 March 2016

Devansh Arpit

Yingbo Zhou

Bhargava U. Kota

V. Govindaraju

ArXiv PDF HTML

Papers citing "Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks"

33 / 33 papers shown

Title
Normalization and effective learning rates in reinforcement learning Clare Lyle Zeyu Zheng Khimya Khetarpal James Martens H. V. Hasselt Razvan Pascanu Will Dabney 24 7 0 01 Jul 2024
Analyzing and Improving the Training Dynamics of Diffusion Models Tero Karras M. Aittala J. Lehtinen Janne Hellsten Timo Aila S. Laine 42 157 0 05 Dec 2023
BCN: Batch Channel Normalization for Image Classification Afifa Khaled Chao Li Jia Ning Kun He 20 6 0 01 Dec 2023
Information Geometrically Generalized Covariate Shift Adaptation Masanari Kimura H. Hino OOD 19 5 0 19 Apr 2023
Noise Injection as a Probe of Deep Learning Dynamics Noam Levi I. Bloch M. Freytsis T. Volansky 40 2 0 24 Oct 2022
Batch Layer Normalization, A new normalization layer for CNNs and RNN A. Ziaee Erion cCano 19 13 0 19 Sep 2022
Revisiting Batch Norm Initialization Jim Davis Logan Frank 22 4 0 26 Oct 2021
"BNN - BN = ?": Training Binary Neural Networks without Batch Normalization Tianlong Chen Zhenyu Zhang Xu Ouyang Zechun Liu Zhiqiang Shen Zhangyang Wang MQ 43 36 0 16 Apr 2021
High-Performance Large-Scale Image Recognition Without Normalization Andrew Brock Soham De Samuel L. Smith Karen Simonyan VLM 223 512 0 11 Feb 2021
Momentum^2 Teacher: Momentum Teacher with Momentum Statistics for Self-Supervised Learning Zeming Li Songtao Liu Jian Sun 51 16 0 19 Jan 2021
A Comprehensive and Modularized Statistical Framework for Gradient Norm Equality in Deep Neural Networks Zhaodong Chen Lei Deng Bangyan Wang Guoqi Li Yuan Xie 35 28 0 01 Jan 2020
Filter Response Normalization Layer: Eliminating Batch Dependence in the Training of Deep Neural Networks Saurabh Singh Shankar Krishnan UQCV 74 125 0 21 Nov 2019
Root Mean Square Layer Normalization Biao Zhang Rico Sennrich 13 665 0 16 Oct 2019
Spectral Regularization for Combating Mode Collapse in GANs Kanglin Liu Wenming Tang Fei Zhou Guoping Qiu GAN DRL 33 81 0 29 Aug 2019
EvalNorm: Estimating Batch Normalization Statistics for Evaluation Saurabh Singh Abhinav Shrivastava 26 51 0 12 Apr 2019
Micro-Batch Training with Batch-Channel Normalization and Weight Standardization Siyuan Qiao Huiyu Wang Chenxi Liu Wei Shen Alan Yuille MQ 32 144 0 25 Mar 2019
Accelerating Training of Deep Neural Networks with a Standardization Loss Jasmine Collins Johannes Ballé Jonathon Shlens 21 3 0 03 Mar 2019
Mode Normalization Lucas Deecke Iain Murray Hakan Bilen OOD 29 33 0 12 Oct 2018
NU-LiteNet: Mobile Landmark Recognition using Convolutional Neural Networks C. Termritthikun S. Kanprachar P. Muneesawang 19 20 0 02 Oct 2018
Decorrelated Batch Normalization Lei Huang Dawei Yang B. Lang Jia Deng 16 190 0 23 Apr 2018
Revisiting Small Batch Training for Deep Neural Networks Dominic Masters Carlo Luschi ODL 37 661 0 20 Apr 2018
Spectral Normalization for Generative Adversarial Networks Takeru Miyato Toshiki Kataoka Masanori Koyama Yuichi Yoshida ODL 44 4,399 0 16 Feb 2018
The exploding gradient problem demystified - definition, prevalence, impact, origin, tradeoffs, and solutions George Philipp D. Song J. Carbonell ODL 35 46 0 15 Dec 2017
Riemannian approach to batch normalization Minhyung Cho Jaehyung Lee 29 94 0 27 Sep 2017
Comparison of Batch Normalization and Weight Normalization Algorithms for the Large-scale Image Classification Igor Gitman Boris Ginsburg 8 65 0 24 Sep 2017
Thinking Fast and Slow with Deep Learning and Tree Search Thomas W. Anthony Zheng Tian David Barber 19 384 0 23 May 2017
Diminishing Batch Normalization Yintai Ma Diego Klabjan 31 15 0 22 May 2017
Reducing Bias in Production Speech Models Eric Battenberg R. Child Adam Coates Christopher Fougner Yashesh Gaur ... Vinay Rao S. Satheesh David Seetapun Anuroop Sriram Zhenyao Zhu 38 10 0 11 May 2017
On the Effects of Batch and Weight Normalization in Generative Adversarial Networks Sitao Xiang Hao Li GAN 35 83 0 13 Apr 2017
Sharp Minima Can Generalize For Deep Nets Laurent Dinh Razvan Pascanu Samy Bengio Yoshua Bengio ODL 46 757 0 15 Mar 2017
All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulation Di Xie Jiang Xiong Shiliang Pu 19 181 0 06 Mar 2017
Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks Chunjie Luo Jianfeng Zhan Lei Wang Qiang Yang 24 199 0 20 Feb 2017
Adding Gradient Noise Improves Learning for Very Deep Networks Arvind Neelakantan Luke Vilnis Quoc V. Le Ilya Sutskever Lukasz Kaiser Karol Kurach James Martens AI4CE ODL 27 541 0 21 Nov 2015