Stochastic Gradient Descent with Nonlinear Conjugate Gradient-Style
Adaptive Momentum

Stochastic Gradient Descent with Nonlinear Conjugate Gradient-Style Adaptive Momentum

3 December 2020

Bao Wang

ArXiv (abs)PDF HTML

Papers citing "Stochastic Gradient Descent with Nonlinear Conjugate Gradient-Style Adaptive Momentum"

10 / 10 papers shown

Title
When Large Language Model Meets Optimization Sen Huang Kaixiang Yang Sheng Qi Rui Wang 102 12 0 16 May 2024
Homotopy Relaxation Training Algorithms for Infinite-Width Two-Layer ReLU Neural Networks Yahong Yang Qipin Chen Wenrui Hao 51 4 0 26 Sep 2023
Proximal Implicit ODE Solvers for Accelerating Learning Neural ODEs Justin Baker Hedi Xia Yiwei Wang E. Cherkaev A. Narayan Long Chen Jack Xin Andrea L. Bertozzi Stanley J. Osher Bao Wang 58 7 0 19 Apr 2022
Learning POD of Complex Dynamics Using Heavy-ball Neural ODEs Justin Baker E. Cherkaev A. Narayan Bao Wang AI4CE 50 5 0 24 Feb 2022
Training Deep Neural Networks with Adaptive Momentum Inspired by the Quadratic Optimization Tao Sun Huaming Ling Zuoqiang Shi Dongsheng Li Bao Wang ODL 65 13 0 18 Oct 2021
How Does Momentum Benefit Deep Neural Networks Architecture Design? A Few Case Studies Bao Wang Hedi Xia T. Nguyen Stanley Osher AI4CE 103 10 0 13 Oct 2021
AIR-Net: Adaptive and Implicit Regularization Neural Network for Matrix Completion Zhemin Li Tao Sun Hongxia Wang Bao Wang 88 6 0 12 Oct 2021
Practical and Fast Momentum-Based Power Methods Tahseen Rabbani Apollo Jain Arjun Rajkumar Furong Huang 18 2 0 20 Aug 2021
Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers Robin M. Schmidt Frank Schneider Philipp Hennig ODL 202 168 0 03 Jul 2020
Adaptive Inertia: Disentangling the Effects of Adaptive Learning Rate and Momentum Zeke Xie Xinrui Wang Huishuai Zhang Issei Sato Masashi Sugiyama ODL 98 47 0 29 Jun 2020