Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks

14 June 2018

Papers citing "Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks"

22 / 22 papers shown

Title
On the Neural Tangent Kernel of Equilibrium Models Zhili Feng J. Zico Kolter 18 6 0 21 Oct 2023
Criticality versus uniformity in deep neural networks A. Bukva Jurriaan de Gier Kevin T. Grosvenor R. Jefferson K. Schalm Eliot Schwander 31 3 0 10 Apr 2023
Global Optimality of Elman-type RNN in the Mean-Field Regime Andrea Agazzi Jian-Xiong Lu Sayan Mukherjee MLT 34 1 0 12 Mar 2023
On the Initialisation of Wide Low-Rank Feedforward Neural Networks Thiziri Nait Saada Jared Tanner 13 1 0 31 Jan 2023
Stretched and measured neural predictions of complex network dynamics V. Vasiliauskaite Nino Antulov-Fantulin 30 1 0 12 Jan 2023
Statistical Physics of Deep Neural Networks: Initialization toward Optimal Channels Kangyu Weng Aohua Cheng Ziyang Zhang Pei Sun Yang Tian 50 2 0 04 Dec 2022
Dynamical Isometry for Residual Networks Advait Gadhikar R. Burkholz ODL AI4CE 40 2 0 05 Oct 2022
Random orthogonal additive filters: a solution to the vanishing/exploding gradient of deep neural networks Andrea Ceni ODL 23 3 0 03 Oct 2022
Recency Dropout for Recurrent Recommender Systems Bo-Yu Chang Can Xu Matt Le Jingchen Feng Ya Le Sriraj Badam Ed H. Chi Minmin Chen 25 3 0 26 Jan 2022
AutoInit: Analytic Signal-Preserving Weight Initialization for Neural Networks G. Bingham Risto Miikkulainen ODL 24 4 0 18 Sep 2021
Towards quantifying information flows: relative entropy in deep neural networks and the renormalization group J. Erdmenger Kevin T. Grosvenor R. Jefferson 54 17 0 14 Jul 2021
Tensor Programs III: Neural Matrix Laws Greg Yang 11 43 0 22 Sep 2020
Beyond Signal Propagation: Is Feature Diversity Necessary in Deep Neural Network Initialization? Yaniv Blumenfeld D. Gilboa Daniel Soudry ODL 27 13 0 02 Jul 2020
Tensor Programs II: Neural Tangent Kernel for Any Architecture Greg Yang 58 134 0 25 Jun 2020
Mean field theory for deep dropout networks: digging up gradient backpropagation deeply Wei Huang R. Xu Weitao Du Yutian Zeng Yunce Zhao 19 6 0 19 Dec 2019
Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes Greg Yang 33 190 0 28 Oct 2019
On the expected behaviour of noise regularised deep neural networks as Gaussian processes Arnu Pretorius Herman Kamper Steve Kroon 16 9 0 12 Oct 2019
Asymptotics of Wide Networks from Feynman Diagrams Ethan Dyer Guy Gur-Ari 24 113 0 25 Sep 2019
On the Convergence Rate of Training Recurrent Neural Networks Zeyuan Allen-Zhu Yuanzhi Li Zhao-quan Song 18 191 0 29 Oct 2018
Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes Roman Novak Lechao Xiao Jaehoon Lee Yasaman Bahri Greg Yang Jiri Hron Daniel A. Abolafia Jeffrey Pennington Jascha Narain Sohl-Dickstein UQCV BDL 25 306 0 11 Oct 2018
Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach Ryo Karakida S. Akaho S. Amari FedML 47 140 0 04 Jun 2018
Learning Unitary Operators with Help From u(n) Stephanie L. Hyland Gunnar Rätsch 97 41 0 17 Jul 2016