Hessian based analysis of SGD for Deep Nets: Dynamics and Generalization

SDM (SDM), 2019

24 July 2019

Papers citing "Hessian based analysis of SGD for Deep Nets: Dynamics and Generalization"

30 / 30 papers shown

Title
$Stacey: Promoting Stochastic Steepest Descent via Accelerated $\ell_p$-Smooth Nonconvex Optimization$ Stacey: Promoting Stochastic Steepest Descent via Accelerated $\ell_p$ -Smooth Nonconvex Optimization Xinyu Luo Cedar Site Bai Bolian Li Petros Drineas Ruqi Zhang Brian Bullins 93 2 0 07 Jun 2025
Towards Quantifying the Hessian Structure of Neural Networks Zhaorui Dong Yushun Zhang Jianfeng Yao Jianfeng Yao 167 2 0 05 May 2025
Sketched Adaptive Federated Deep Learning: A Sharp Convergence Analysis Zhijie Chen Qiaobo Li A. Banerjee FedML 284 0 0 11 Nov 2024
Loss Gradient Gaussian Width based Generalization and Optimization Guarantees A. Banerjee Qiaobo Li Yingxue Zhou 275 0 0 11 Jun 2024
PCDP-SGD: Improving the Convergence of Differentially Private SGD via Projection in Advance Haichao Sha Ruixuan Liu Yi-xiao Liu Hong Chen 191 2 0 06 Dec 2023
Taxonomy Adaptive Cross-Domain Adaptation in Medical Imaging via Optimization Trajectory DistillationIEEE International Conference on Computer Vision (ICCV), 2023 Jianan Fan Dongnan Liu Hang Chang Heng-Chiao Huang Mei Chen Weidong (Tom) Cai OOD 123 10 0 27 Jul 2023
Choosing Public Datasets for Private Machine Learning via Gradient Subspace Distance Xin Gu Gautam Kamath Zhiwei Steven Wu 188 19 0 02 Mar 2023
Escaping Saddle Points for Effective Generalization on Class-Imbalanced DataNeural Information Processing Systems (NeurIPS), 2022 Harsh Rangwani Sumukh K Aithal Mayank Mishra R. Venkatesh Babu 154 36 0 28 Dec 2022
On the Overlooked Structure of Stochastic GradientsNeural Information Processing Systems (NeurIPS), 2022 Zeke Xie Qian-Yuan Tang Mingming Sun P. Li 168 7 0 05 Dec 2022
Two Facets of SDE Under an Information-Theoretic Lens: Generalization of SGD via Training Trajectories and via Terminal StatesConference on Uncertainty in Artificial Intelligence (UAI), 2022 Ziqiao Wang Yongyi Mao 190 12 0 19 Nov 2022
Noise Injection as a Probe of Deep Learning Dynamics Noam Levi I. Bloch M. Freytsis T. Volansky 122 3 0 24 Oct 2022
FIT: A Metric for Model SensitivityInternational Conference on Learning Representations (ICLR), 2022 Ben Zandonati Adrian Alan Pol M. Pierini Olya Sirkin Tal Kopetz MQ 131 9 0 16 Oct 2022
Analysis of Branch Specialization and its Application in Image Decomposition Jonathan Brokman Guy Gilboa 82 2 0 12 Jun 2022
Improving Differentially Private SGD via Randomly Sparsified Gradients Junyi Zhu Matthew B. Blaschko 184 7 0 01 Dec 2021
Fishr: Invariant Gradient Variances for Out-of-Distribution GeneralizationInternational Conference on Machine Learning (ICML), 2021 Alexandre Ramé Corentin Dancette Matthieu Cord OOD 238 239 0 07 Sep 2021
Large Scale Private Learning via Low-rank ReparametrizationInternational Conference on Machine Learning (ICML), 2021 Da Yu Huishuai Zhang Wei Chen Jian Yin Tie-Yan Liu 180 114 0 17 Jun 2021
Vanishing Curvature and the Power of Adaptive Methods in Randomly Initialized Deep Networks Antonio Orvieto Jonas Köhler Dario Pavllo Thomas Hofmann Aurelien Lucchi ODL 88 6 0 07 Jun 2021
Privately Learning SubspacesNeural Information Processing Systems (NeurIPS), 2021 Vikrant Singhal Thomas Steinke 257 22 0 28 May 2021
Empirically explaining SGD from a line search perspectiveInternational Conference on Artificial Neural Networks (ICANN), 2021 Max Mutschler A. Zell ODL LRM 140 4 0 31 Mar 2021
Hessian Eigenspectra of More Realistic Nonlinear ModelsNeural Information Processing Systems (NeurIPS), 2021 Zhenyu Liao Michael W. Mahoney 169 37 0 02 Mar 2021
Gradient Descent on Neural Networks Typically Occurs at the Edge of StabilityInternational Conference on Learning Representations (ICLR), 2021 Jeremy M. Cohen Simran Kaur Yuanzhi Li J. Zico Kolter Ameet Talwalkar ODL 253 325 0 26 Feb 2021
Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private LearningInternational Conference on Learning Representations (ICLR), 2021 Da Yu Huishuai Zhang Wei Chen Tie-Yan Liu FedML SILM 263 125 0 25 Feb 2021
Provable Super-Convergence with a Large Cyclical Learning RateIEEE Signal Processing Letters (IEEE SPL), 2021 Samet Oymak 113 13 0 22 Feb 2021
Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networks Yikai Wu Xingyu Zhu Chenwei Wu Annie Wang Rong Ge 369 45 0 08 Oct 2020
A Framework for Private Matrix Analysis Jalaj Upadhyay Sarvagya Upadhyay 148 4 0 06 Sep 2020
Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra Vardan Papyan 129 83 0 27 Aug 2020
Bypassing the Ambient Dimension: Private SGD with Gradient Subspace IdentificationInternational Conference on Learning Representations (ICLR), 2020 Yingxue Zhou Zhiwei Steven Wu A. Banerjee 143 116 0 07 Jul 2020
De-randomized PAC-Bayes Margin Bounds: Applications to Non-convex and Non-smooth Predictors A. Banerjee Tiancong Chen Yingxue Zhou BDL 165 9 0 23 Feb 2020
The Geometry of Sign Gradient Descent Lukas Balles Fabian Pedregosa Nicolas Le Roux ODL 114 33 0 19 Feb 2020
Dissecting Non-Vacuous Generalization Bounds based on the Mean-Field ApproximationInternational Conference on Machine Learning (ICML), 2019 Konstantinos Pitas 126 8 0 06 Sep 2019