Characterizing signal propagation to close the performance gap in unnormalized ResNets

21 January 2021

Papers citing "Characterizing signal propagation to close the performance gap in unnormalized ResNets"

32 / 32 papers shown

Title
Analyzing and Improving the Training Dynamics of Diffusion Models Tero Karras M. Aittala J. Lehtinen Janne Hellsten Timo Aila S. Laine 28 153 0 05 Dec 2023
Convex Dual Theory Analysis of Two-Layer Convolutional Neural Networks with Soft-Thresholding Chunyan Xiong Meng Lu Xiaotong Yu JIAN-PENG Cao Zhong Chen D. Guo X. Qu MLT 35 0 0 14 Apr 2023
Inductive biases in deep learning models for weather prediction Jannik Thümmel Matthias Karlbauer S. Otte C. Zarfl Georg Martius ... Thomas Scholten Ulrich Friedrich V. Wulfmeyer B. Goswami Martin Volker Butz AI4CE 38 5 0 06 Apr 2023
Graph Mixture of Experts: Learning on Large-Scale Graphs with Explicit Diversity Modeling Haotao Wang Ziyu Jiang Yuning You Yan Han Gaowen Liu Jayanth Srinivasa Ramana Rao Kompella Zhangyang Wang 21 28 0 06 Apr 2023
How to Use Dropout Correctly on Residual Networks with Batch Normalization Bum Jun Kim Hyeyeon Choi Hyeonah Jang Donggeon Lee Sang Woo Kim 22 7 0 13 Feb 2023
Binarized Neural Machine Translation Yichi Zhang Ankush Garg Yuan Cao Lukasz Lew Behrooz Ghorbani Zhiru Zhang Orhan Firat MQ 29 14 0 09 Feb 2023
Out of Distribution Performance of State of Art Vision Model Salman Rahman W. Lee 26 2 0 25 Jan 2023
Differentially Private Image Classification from Features Harsh Mehta Walid Krichene Abhradeep Thakurta Alexey Kurakin Ashok Cutkosky 46 7 0 24 Nov 2022
FusionFormer: Fusing Operations in Transformer for Efficient Streaming Speech Recognition Xingcheng Song Di Wu Binbin Zhang Zhiyong Wu Wenpeng Li ... Peng Zhang Zhendong Peng Fuping Pan Changbao Zhu Zhongqin Wu 19 2 0 31 Oct 2022
Online Training Through Time for Spiking Neural Networks Mingqing Xiao Qingyan Meng Zongpeng Zhang D.K. He Zhouchen Lin 45 60 0 09 Oct 2022
Dynamical Isometry for Residual Networks Advait Gadhikar R. Burkholz ODL AI4CE 32 2 0 05 Oct 2022
Symmetry Regularization and Saturating Nonlinearity for Robust Quantization Sein Park Yeongsang Jang Eunhyeok Park MQ 14 1 0 31 Jul 2022
Removing Batch Normalization Boosts Adversarial Training Haotao Wang Aston Zhang Shuai Zheng Xingjian Shi Mu Li Zhangyang Wang 32 41 0 04 Jul 2022
Adapting the Linearised Laplace Model Evidence for Modern Deep Learning Javier Antorán David Janz J. Allingham Erik A. Daxberger Riccardo Barbano Eric T. Nalisnick José Miguel Hernández-Lobato UQCV BDL 27 28 0 17 Jun 2022
Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction Kaifeng Lyu Zhiyuan Li Sanjeev Arora FAtt 37 69 0 14 Jun 2022
Scaling ResNets in the Large-depth Regime P. Marion Adeline Fermanian Gérard Biau Jean-Philippe Vert 26 16 0 14 Jun 2022
Guidelines for the Regularization of Gammas in Batch Normalization for Deep Residual Networks Bum Jun Kim Hyeyeon Choi Hyeonah Jang Dong Gu Lee Wonseok Jeong Sang Woo Kim 11 4 0 15 May 2022
Online Convolutional Re-parameterization Mu Hu Junyi Feng Jiashen Hua Baisheng Lai Jianqiang Huang Xiaojin Gong Xiansheng Hua 16 26 0 02 Apr 2022
Continual Normalization: Rethinking Batch Normalization for Online Continual Learning Quang-Cuong Pham Chenghao Liu S. Hoi BDL OnRL 28 57 0 30 Mar 2022
Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifiers Guodong Zhang Aleksandar Botev James Martens OffRL 21 26 0 15 Mar 2022
Regularising for invariance to data augmentation improves supervised learning Aleksander Botev Matthias Bauer Soham De 30 14 0 07 Mar 2022
Towards Learning Universal Audio Representations Luyu Wang Pauline Luc Yan Wu Adrià Recasens Lucas Smaira ... Andrew Jaegle Jean-Baptiste Alayrac Sander Dieleman João Carreira Aaron van den Oord SSL 24 68 0 23 Nov 2021
Swin Transformer V2: Scaling Up Capacity and Resolution Ze Liu Han Hu Yutong Lin Zhuliang Yao Zhenda Xie ... Yue Cao Zheng-Wei Zhang Li Dong Furu Wei B. Guo ViT 49 1,746 0 18 Nov 2021
AutoInit: Analytic Signal-Preserving Weight Initialization for Neural Networks G. Bingham Risto Miikkulainen ODL 24 4 0 18 Sep 2021
GemNet: Universal Directional Graph Neural Networks for Molecules Johannes Klicpera Florian Becker Stephan Günnemann AI4CE 21 433 0 02 Jun 2021
"BNN - BN = ?": Training Binary Neural Networks without Batch Normalization Tianlong Chen Zhenyu (Allen) Zhang Xu Ouyang Zechun Liu Zhiqiang Shen Zhangyang Wang MQ 33 36 0 16 Apr 2021
Going deeper with Image Transformers Hugo Touvron Matthieu Cord Alexandre Sablayrolles Gabriel Synnaeve Hervé Jégou ViT 25 986 0 31 Mar 2021
GradInit: Learning to Initialize Neural Networks for Stable and Efficient Training Chen Zhu Renkun Ni Zheng Xu Kezhi Kong W. R. Huang Tom Goldstein ODL 31 53 0 16 Feb 2021
High-Performance Large-Scale Image Recognition Without Normalization Andrew Brock Soham De Samuel L. Smith Karen Simonyan VLM 223 512 0 11 Feb 2021
Bag of Tricks for Image Classification with Convolutional Neural Networks Tong He Zhi-Li Zhang Hang Zhang Zhongyue Zhang Junyuan Xie Mu Li 221 1,399 0 04 Dec 2018
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications Andrew G. Howard Menglong Zhu Bo Chen Dmitry Kalenichenko Weijun Wang Tobias Weyand M. Andreetto Hartwig Adam 3DH 950 20,561 0 17 Apr 2017
Aggregated Residual Transformations for Deep Neural Networks Saining Xie Ross B. Girshick Piotr Dollár Z. Tu Kaiming He 297 10,216 0 16 Nov 2016