Dynamical Isometry is Achieved in Residual Networks in a Universal Way
for any Activation Function

Dynamical Isometry is Achieved in Residual Networks in a Universal Way for any Activation Function

24 September 2018

Stanislaw Jastrzebski

Papers citing "Dynamical Isometry is Achieved in Residual Networks in a Universal Way for any Activation Function"

14 / 14 papers shown

Title
Dynamical Isometry for Residual Networks Advait Gadhikar R. Burkholz ODL AI4CE 45 2 0 05 Oct 2022
Entangled Residual Mappings Mathias Lechner Ramin Hasani Z. Babaiee Radu Grosu Daniela Rus T. Henzinger Sepp Hochreiter 19 5 0 02 Jun 2022
Neural Born Iteration Method For Solving Inverse Scattering Problems: 2D Cases Tao Shan Zhichao Lin Xiaoqian Song Maokun Li Fan Yang Zhensheng Xu 48 18 0 18 Dec 2021
A Johnson--Lindenstrauss Framework for Randomly Initialized CNNs Ido Nachum Jan Hkazla Michael C. Gastpar Anatoly Khina 36 0 0 03 Nov 2021
On Random Matrices Arising in Deep Neural Networks: General I.I.D. Case L. Pastur V. Slavin CML 24 12 0 20 Nov 2020
Tensor Programs III: Neural Matrix Laws Greg Yang 19 45 0 22 Sep 2020
Deep Isometric Learning for Visual Recognition Haozhi Qi Chong You Xinyu Wang Yi Ma Jitendra Malik VLM 35 54 0 30 Jun 2020
The Spectrum of Fisher Information of Deep Networks Achieving Dynamical Isometry Tomohiro Hayase Ryo Karakida 29 7 0 14 Jun 2020
A Comprehensive and Modularized Statistical Framework for Gradient Norm Equality in Deep Neural Networks Zhaodong Chen Lei Deng Bangyan Wang Guoqi Li Yuan Xie 40 28 0 01 Jan 2020
Optimization for deep learning: theory and algorithms Ruoyu Sun ODL 30 168 0 19 Dec 2019
A Signal Propagation Perspective for Pruning Neural Networks at Initialization Namhoon Lee Thalaiyasingam Ajanthan Stephen Gould Philip Torr AAML 38 152 0 14 Jun 2019
Residual Networks as Nonlinear Systems: Stability Analysis using Linearization Kai Rothauge Z. Yao Zixi Hu Michael W. Mahoney 15 2 0 31 May 2019
Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks Lechao Xiao Yasaman Bahri Jascha Narain Sohl-Dickstein S. Schoenholz Jeffrey Pennington 244 350 0 14 Jun 2018
The Loss Surfaces of Multilayer Networks A. Choromańska Mikael Henaff Michaël Mathieu Gerard Ben Arous Yann LeCun ODL 186 1,186 0 30 Nov 2014