v1v2 (latest)

How to Initialize your Network? Robust Initialization for WeightNorm & ResNets

5 June 2019

Papers citing "How to Initialize your Network? Robust Initialization for WeightNorm & ResNets"

30 / 30 papers shown

Title
FINE: Factorizing Knowledge for Initialization of Variable-sized Diffusion Models Yucheng Xie Fu Feng Ruixiao Shi Jing Wang Xin Geng AI4CE 62 3 0 28 Sep 2024
On the consistency of hyper-parameter selection in value-based deep reinforcement learning J. Obando-Ceron J. G. Araújo Rameswar Panda Pablo Samuel Castro 118 9 0 25 Jun 2024
Deep linear networks for regression are implicitly regularized towards flat minima Pierre Marion Lénaic Chizat ODL 104 6 0 22 May 2024
Exploring Learngene via Stage-wise Weight Sharing for Initializing Variable-sized Models Shiyu Xia Wenxuan Zhu Xu Yang Xin Geng 54 2 0 25 Apr 2024
Neural Echos: Depthwise Convolutional Filters Replicate Biological Receptive Fields Z. Babaiee Peyman M. Kiasari Daniela Rus Radu Grosu MDE 26 3 0 18 Jan 2024
Simplifying Transformer Blocks Bobby He Thomas Hofmann 109 36 0 03 Nov 2023
Expressive Monotonic Neural Networks O. Kitouni Niklas Nolte Mike Williams 37 7 0 14 Jul 2023
NeuBTF: Neural fields for BTF encoding and transfer Carlos Rodriguez-Pardo Konstantinos Kazatzis Jorge López-Moreno Elena Garces AI4CE 84 9 0 03 Jul 2023
Initial Guessing Bias: How Untrained Networks Favor Some Classes Emanuele Francazi Aurelien Lucchi Marco Baity-Jesi AI4CE 75 4 0 01 Jun 2023
Spiking Network Initialisation and Firing Rate Collapse Nicolas Perez Nieves Dan F. M. Goodman 89 2 0 13 May 2023
Multi-task neural networks by learned contextual inputs Anders T. Sandnes B. Grimstad O. Kolbjørnsen 51 1 0 01 Mar 2023
Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagation Bobby He James Martens Guodong Zhang Aleksandar Botev Andy Brock Samuel L. Smith Yee Whye Teh 85 30 0 20 Feb 2023
On the Lipschitz Constant of Deep Networks and Double Descent Matteo Gamba Hossein Azizpour Mårten Björkman 98 7 0 28 Jan 2023
Padding Module: Learning the Padding in Deep Neural Networks Fahad Alrasheedi Agnibh Dasgupta Pei-Chi Huang KELM VLM 27 17 0 11 Jan 2023
Eigenvalue initialisation and regularisation for Koopman autoencoders Jack W. Miller Charles OÑeill N. Constantinou Omri Azencot 57 2 0 23 Dec 2022
Deep Double Descent via Smooth Interpolation Matteo Gamba Erik Englesson Mårten Björkman Hossein Azizpour 169 11 0 21 Sep 2022
When Bioprocess Engineering Meets Machine Learning: A Survey from the Perspective of Automated Bioprocess Development Nghia Duong-Trung Stefan Born Jong Woo Kim M. Schermeyer Katharina Paulick ... Thorben Werner Randolf Scholz Lars Schmidt-Thieme Peter Neubauer Ernesto Martinez 71 20 0 02 Sep 2022
Scaling ResNets in the Large-depth Regime Pierre Marion Adeline Fermanian Gérard Biau Jean-Philippe Vert 105 16 0 14 Jun 2022
Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse Lorenzo Noci Sotiris Anagnostidis Luca Biggio Antonio Orvieto Sidak Pal Singh Aurelien Lucchi 105 75 0 07 Jun 2022
Entangled Residual Mappings Mathias Lechner Ramin Hasani Z. Babaiee Radu Grosu Daniela Rus T. Henzinger Sepp Hochreiter 78 5 0 02 Jun 2022
A Robust Initialization of Residual Blocks for Effective ResNet Training without Batch Normalization Enrico Civitelli Alessio Sortino Matteo Lapucci Francesco Bagattini G. Galvan OffRL ODL OOD 32 1 0 23 Dec 2021
Robust and Provably Monotonic Networks O. Kitouni Niklas Nolte Mike Williams 38 10 0 30 Nov 2021
AutoInit: Analytic Signal-Preserving Weight Initialization for Neural Networks G. Bingham Risto Miikkulainen ODL 67 4 0 18 Sep 2021
A Weight Initialization Based on the Linear Product Structure for Neural Networks Qipin Chen Wenrui Hao Juncai He 8 8 0 01 Sep 2021
The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initialization Mufan Li Mihai Nica Daniel M. Roy 113 34 0 07 Jun 2021
Fast Certified Robust Training with Short Warmup Zhouxing Shi Yihan Wang Huan Zhang Jinfeng Yi Cho-Jui Hsieh AAML 99 57 0 31 Mar 2021
Training Deep Neural Networks Without Batch Normalization D. Gaur Joachim Folz Andreas Dengel ODL 36 10 0 18 Aug 2020
Optimization Theory for ReLU Neural Networks Trained with Normalization Layers Yonatan Dukler Quanquan Gu Guido Montúfar 72 30 0 11 Jun 2020
Revisiting Initialization of Neural Networks Maciej Skorski Alessandro Temperoni Martin Theobald 31 2 0 20 Apr 2020
Stabilize Deep ResNet with A Sharp Scaling Factor $τ$ Huishuai Zhang Da Yu Mingyang Yi Wei Chen Tie-Yan Liu 57 9 0 17 Mar 2019