Communities
Connect sessions
AI calendar
Organizations
Join Slack
Contact Sales
Search
Open menu
Home
Papers
1901.08987
Cited By
v1
v2 (latest)
Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs
25 January 2019
D. Gilboa
B. Chang
Minmin Chen
Greg Yang
S. Schoenholz
Ed H. Chi
Jeffrey Pennington
Re-assign community
ArXiv (abs)
PDF
HTML
Papers citing
"Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs"
30 / 30 papers shown
Time-Scale Coupling Between States and Parameters in Recurrent Neural Networks
Lorenzo Livi
211
1
0
16 Aug 2025
Revisiting Glorot Initialization for Long-Range Linear Recurrences
Noga Bar
Mariia Seleznova
Yotam Alexander
Gitta Kutyniok
Raja Giryes
172
0
0
26 May 2025
Deep Neural Network Initialization with Sparsity Inducing Activations
Ilan Price
Nicholas Daultry Ball
Samuel C.H. Lam
Adam C. Jones
Jared Tanner
AI4CE
189
2
0
25 Feb 2024
Gradient Flossing: Improving Gradient Descent through Dynamic Control of Jacobians
Rainer Engelken
233
10
0
28 Dec 2023
On the Neural Tangent Kernel of Equilibrium Models
Zhili Feng
J. Zico Kolter
221
9
0
21 Oct 2023
On the Initialisation of Wide Low-Rank Feedforward Neural Networks
Thiziri Nait Saada
Jared Tanner
179
2
0
31 Jan 2023
Statistical Physics of Deep Neural Networks: Initialization toward Optimal Channels
Physical Review Research (Phys. Rev. Res.), 2022
Kangyu Weng
Aohua Cheng
Ziyang Zhang
Pei Sun
Yang Tian
283
5
0
04 Dec 2022
Analysis of Convolutions, Non-linearity and Depth in Graph Neural Networks using Neural Tangent Kernel
Mahalakshmi Sabanayagam
Pascal Esser
Debarghya Ghoshdastidar
382
4
0
18 Oct 2022
Random orthogonal additive filters: a solution to the vanishing/exploding gradient of deep neural networks
IEEE Transactions on Neural Networks and Learning Systems (TNNLS), 2022
Andrea Ceni
ODL
156
12
0
03 Oct 2022
Generalizing Goal-Conditioned Reinforcement Learning with Variational Causal Reasoning
Neural Information Processing Systems (NeurIPS), 2022
Wenhao Ding
Haohong Lin
Yue Liu
Ding Zhao
LRM
545
50
0
19 Jul 2022
Recency Dropout for Recurrent Recommender Systems
Bo-Yu Chang
Can Xu
Matt Le
Jingchen Feng
Ya Le
Sriraj Badam
Ed H. Chi
Minmin Chen
122
5
0
26 Jan 2022
The edge of chaos: quantum field theory and deep neural networks
SciPost Physics (SciPost Phys.), 2021
Kevin T. Grosvenor
R. Jefferson
212
28
0
27 Sep 2021
Towards quantifying information flows: relative entropy in deep neural networks and the renormalization group
SciPost Physics (SciPost Phys.), 2021
J. Erdmenger
Kevin T. Grosvenor
R. Jefferson
169
23
0
14 Jul 2021
Asymptotic Freeness of Layerwise Jacobians Caused by Invariance of Multilayer Perceptron: The Haar Orthogonal Case
Communications in Mathematical Physics (Commun. Math. Phys.), 2021
B. Collins
Tomohiro Hayase
243
8
0
24 Mar 2021
Feature Learning in Infinite-Width Neural Networks
Greg Yang
J. E. Hu
MLT
422
181
0
30 Nov 2020
Beyond Signal Propagation: Is Feature Diversity Necessary in Deep Neural Network Initialization?
Yaniv Blumenfeld
D. Gilboa
Daniel Soudry
ODL
199
16
0
02 Jul 2020
On Lyapunov Exponents for RNNs: Understanding Information Propagation Using Dynamical Systems Tools
Frontiers in Applied Mathematics and Statistics (FAMS), 2020
Ryan H. Vogt
M. P. Touzel
Eli Shlizerman
Guillaume Lajoie
213
53
0
25 Jun 2020
The Spectrum of Fisher Information of Deep Networks Achieving Dynamical Isometry
International Conference on Artificial Intelligence and Statistics (AISTATS), 2020
Tomohiro Hayase
Ryo Karakida
298
9
0
14 Jun 2020
Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classification
Neural Information Processing Systems (NeurIPS), 2020
Francesca Mignacco
Florent Krzakala
Pierfrancesco Urbani
Lenka Zdeborová
MLT
346
73
0
10 Jun 2020
ReZero is All You Need: Fast Convergence at Large Depth
Conference on Uncertainty in Artificial Intelligence (UAI), 2020
Thomas C. Bachlechner
Bodhisattwa Prasad Majumder
H. H. Mao
G. Cottrell
Julian McAuley
AI4CE
379
329
0
10 Mar 2020
Gating creates slow modes and controls phase-space complexity in GRUs and LSTMs
Mathematical and Scientific Machine Learning (MSML), 2020
T. Can
K. Krishnamurthy
D. Schwab
AI4CE
377
20
0
31 Jan 2020
Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks
International Conference on Learning Representations (ICLR), 2020
Wei Hu
Lechao Xiao
Jeffrey Pennington
210
129
0
16 Jan 2020
Disentangling Trainability and Generalization in Deep Neural Networks
Lechao Xiao
Jeffrey Pennington
S. Schoenholz
199
34
0
30 Dec 2019
Mean field theory for deep dropout networks: digging up gradient backpropagation deeply
European Conference on Artificial Intelligence (ECAI), 2019
Wei Huang
R. Xu
Weitao Du
Yutian Zeng
Yunce Zhao
157
6
0
19 Dec 2019
Optimization for deep learning: theory and algorithms
Tian Ding
ODL
340
178
0
19 Dec 2019
One-Shot Pruning of Recurrent Neural Networks by Jacobian Spectrum Evaluation
International Conference on Learning Representations (ICLR), 2019
Matthew Shunshi Zhang
Bradly C. Stadie
121
34
0
30 Nov 2019
Mean-field inference methods for neural networks
Marylou Gabrié
AI4CE
360
36
0
03 Nov 2019
Deep Learning Theory Review: An Optimal Control and Dynamical Systems Perspective
Guan-Horng Liu
Evangelos A. Theodorou
AI4CE
300
74
0
28 Aug 2019
A Mean Field Theory of Quantized Deep Networks: The Quantization-Depth Trade-Off
Neural Information Processing Systems (NeurIPS), 2019
Yaniv Blumenfeld
D. Gilboa
Daniel Soudry
MQ
195
14
0
03 Jun 2019
A Mean Field Theory of Batch Normalization
Greg Yang
Jeffrey Pennington
Vinay Rao
Jascha Narain Sohl-Dickstein
S. Schoenholz
196
185
0
21 Feb 2019
1
