All Papers

0 / 0 papers shown

Title

v1v2 (latest)

Quadratic Suffices for Over-parametrization via Matrix Chernoff Bound

9 June 2019

Zhao Song

Xin Yang

ArXiv (abs)PDF HTML

Papers citing "Quadratic Suffices for Over-parametrization via Matrix Chernoff Bound"

50 / 75 papers shown

Title
Data Uniformity Improves Training Efficiency and More, with a Convergence Framework Beyond the NTK Regime Yuqing Wang Shangding Gu 93 0 0 30 Jun 2025
Evaluating the design space of diffusion-based generative modelsNeural Information Processing Systems (NeurIPS), 2024 Yuqing Wang Ye He Molei Tao DiffM 215 14 0 18 Jun 2024
Approximation and Gradient Descent Training with Neural Networks G. Welper 113 2 0 19 May 2024
Analyzing the Neural Tangent Kernel of Periodically Activated Coordinate Networks Hemanth Saratchandran Shin-Fang Chng Simon Lucey 156 2 0 07 Feb 2024
A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative ModelsAnnual Review of Statistics and Its Application (ARSIA), 2024 Namjoon Suh Guang Cheng MedIm 208 17 0 14 Jan 2024
An Automatic Learning Rate Schedule Algorithm for Achieving Faster Convergence and Steeper Descent Zhao Song Chiwun Yang 148 10 0 17 Oct 2023
On the Convergence of Federated Averaging under Partial Participation for Over-parameterized Neural Networks Xin Liu Wei Tao Dazhi Zhan Yu Pan Xin Ma Yu Ding Zhisong Pan FedML 147 0 0 09 Oct 2023
How many Neurons do we need? A refined Analysis for Shallow Networks trained with Gradient DescentJournal of Statistical Planning and Inference (JSPI), 2023 Mike Nguyen Nicole Mücke MLT 149 6 0 14 Sep 2023
Approximation Results for Gradient Descent trained Neural Networks G. Welper 98 1 0 09 Sep 2023
Six Lectures on Linearized Neural NetworksJournal of Statistical Mechanics: Theory and Experiment (J. Stat. Mech.), 2023 Theodor Misiakiewicz Andrea Montanari 224 16 0 25 Aug 2023
How to Protect Copyright Data in Optimization of Large Language Models?AAAI Conference on Artificial Intelligence (AAAI), 2023 T. Chu Zhao Song Chiwun Yang 136 36 0 23 Aug 2023
Convergence of Two-Layer Regression with Nonlinear Units Yichuan Deng Zhao Song Shenghao Xie 143 8 0 16 Aug 2023
Memory capacity of two layer neural networks with smooth activationsSIAM Journal on Mathematics of Data Science (SIMODS), 2023 Liam Madden Christos Thrampoulidis MLT 212 6 0 03 Aug 2023
Controlling the Inductive Bias of Wide Neural Networks by Modifying the Kernel's Spectrum Amnon Geifman Daniel Barzilai Ronen Basri Meirav Galun 192 9 0 26 Jul 2023
Understanding Deep Neural Networks via Linear Separability of Hidden Layers Chao Zhang Xinyuan Chen Wensheng Li Lixue Liu Wei Wu Dacheng Tao 128 4 0 26 Jul 2023
Efficient SGD Neural Network Training via Sublinear Activated Neuron IdentificationBigData Congress [Services Society] (BSS), 2023 Lianke Qin Zhao Song Yuanyuan Yang 118 9 0 13 Jul 2023
Query Complexity of Active Learning for Function Family With Nearly Orthogonal Basis Xiangyi Chen Zhao Song Baochen Sun Junze Yin Danyang Zhuo 150 4 0 06 Jun 2023
Efficient Asynchronize Stochastic Gradient Algorithm with Structured Data Zhao Song Mingquan Ye 143 4 0 13 May 2023
An Over-parameterized Exponential Regression Yeqi Gao Sridhar Mahadevan Zhao Song 132 42 0 29 Mar 2023
Streaming Kernel PCA Algorithm With Small Space Yichuan Deng Zhao Song Zifan Wang Hangke Zhang 156 4 0 08 Mar 2023
Pruning Before Training May Improve Generalization, Provably Hongru Yang Yingbin Liang Xiaojie Guo Lingfei Wu Zhangyang Wang MLT 163 2 0 01 Jan 2023
Improved Convergence Guarantees for Shallow Neural Networks A. Razborov ODL 149 1 0 05 Dec 2022
Bypass Exponential Time Preprocessing: Fast Neural Network Training via Weight-Data Correlation PreprocessingNeural Information Processing Systems (NeurIPS), 2022 Josh Alman Jiehao Liang Zhao Song Ruizhe Zhang Danyang Zhuo 222 31 0 25 Nov 2022
Finite Sample Identification of Wide Shallow Neural Networks with Biases M. Fornasier T. Klock Marco Mondelli Michael Rauchensteiner 117 6 0 08 Nov 2022
Approximation results for Gradient Descent trained Shallow Neural Networks in $1d$ R. Gentile G. Welper ODL 158 9 0 17 Sep 2022
Neural Tangent Kernel: A Survey Eugene Golikov Eduard Pokonechnyy Vladimir Korviakov 150 17 0 29 Aug 2022
A Sublinear Adversarial Training AlgorithmInternational Conference on Learning Representations (ICLR), 2022 Yeqi Gao Lianke Qin Zhao Song Yitan Wang GAN 131 26 0 10 Aug 2022
Training Overparametrized Neural Networks in Sublinear Time Yichuan Deng Han Hu Zhao Song Omri Weinstein Danyang Zhuo BDL 157 28 0 09 Aug 2022
Provable Acceleration of Nesterov's Accelerated Gradient Method over Heavy Ball Method in Training Over-Parameterized Neural NetworksInternational Joint Conference on Artificial Intelligence (IJCAI), 2022 Xin Liu Wei Tao Wei Li Dazhi Zhan Jun Wang Zhisong Pan ODL 235 1 0 08 Aug 2022
Federated Adversarial Learning: A Framework with Convergence AnalysisInternational Conference on Machine Learning (ICML), 2022 Xiaoxiao Li Zhao Song Jiaming Yang FedML 158 28 0 07 Aug 2022
Bounding the Width of Neural Networks via Coupled Initialization -- A Worst Case AnalysisInternational Conference on Machine Learning (ICML), 2022 Alexander Munteanu Simon Omlor Zhao Song David P. Woodruff 138 16 0 26 Jun 2022
Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterizationNeural Information Processing Systems (NeurIPS), 2022 Simone Bombari Mohammad Hossein Amani Marco Mondelli 136 33 0 20 May 2022
Implicit Bias of MSE Gradient Optimization in Underparameterized Neural NetworksInternational Conference on Learning Representations (ICLR), 2022 Benjamin Bowman Guido Montúfar 142 13 0 12 Jan 2022
Training Multi-Layer Over-Parametrized Neural Network in Subquadratic Time Zhao Song Licheng Zhang Ruizhe Zhang 204 67 0 14 Dec 2021
On the Convergence of Shallow Neural Network Training with Randomly Masked Neurons Fangshuo Liao Anastasios Kyrillidis 217 16 0 05 Dec 2021
Fast Graph Neural Tangent Kernel via Kronecker SketchingAAAI Conference on Artificial Intelligence (AAAI), 2021 Shunhua Jiang Yunze Man Zhao Song Zheng Yu Danyang Zhuo 155 8 0 04 Dec 2021
Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models Tri Dao Beidi Chen Kaizhao Liang Jiaming Yang Zhao Song Atri Rudra Christopher Ré 261 87 0 30 Nov 2021
Subquadratic Overparameterization for Shallow Neural NetworksNeural Information Processing Systems (NeurIPS), 2021 Chaehwan Song Ali Ramezani-Kebrya Thomas Pethick Armin Eftekhari Volkan Cevher 135 33 0 02 Nov 2021
Does Preprocessing Help Training Over-parameterized Neural Networks?Neural Information Processing Systems (NeurIPS), 2021 Zhao Song Shuo Yang Ruizhe Zhang 170 50 0 09 Oct 2021
Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networks Zhichao Wang Yizhe Zhu 185 23 0 20 Sep 2021
Fast Sketching of Polynomial Kernels of Polynomial Degree Zhao Song David P. Woodruff Zheng Yu Lichen Zhang 146 46 0 21 Aug 2021
Provable Convergence of Nesterov's Accelerated Gradient Method for Over-Parameterized Neural Networks Xin Liu Zhisong Pan Wei Tao 215 9 0 05 Jul 2021
Early-stopped neural networks are consistentNeural Information Processing Systems (NeurIPS), 2021 Ziwei Ji Justin D. Li Matus Telgarsky 166 45 0 10 Jun 2021
FL-NTK: A Neural Tangent Kernel-based Framework for Federated Learning Convergence Analysis Baihe Huang Xiaoxiao Li Zhao Song Xin Yang FedML 107 16 0 11 May 2021
GIST: Distributed Training for Large-Scale Graph Convolutional NetworksJournal of Applied and Computational Topology (JACT), 2021 Cameron R. Wolfe Jingkang Yang Arindam Chowdhury Chen Dun Artun Bayer Santiago Segarra Anastasios Kyrillidis BDL GNN LRM 168 11 0 20 Feb 2021
On the Proof of Global Convergence of Gradient Descent for Deep ReLU Networks with Linear WidthsInternational Conference on Machine Learning (ICML), 2021 Quynh N. Nguyen 165 51 0 24 Jan 2021
A Convergence Theory Towards Practical Over-parameterized Deep Neural Networks Asaf Noy Yi Tian Xu Y. Aflalo Lihi Zelnik-Manor Rong Jin 173 3 0 12 Jan 2021
Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU NetworksInternational Conference on Machine Learning (ICML), 2020 Quynh N. Nguyen Marco Mondelli Guido Montúfar 328 92 0 21 Dec 2020
Metric Transforms and Low Rank Matrices via Representation Theory of the Real Hyperrectangle Josh Alman T. Chu Gary Miller Shyam Narayanan Mark Sellke Zhao Song 90 1 0 23 Nov 2020
Algorithms and Hardness for Linear Algebra on Geometric Graphs Josh Alman T. Chu Aaron Schild Zhao Song 200 31 0 04 Nov 2020

v1v2 (latest)

Quadratic Suffices for Over-parametrization via Matrix Chernoff Bound

9 June 2019

Zhao Song

Xin Yang

ArXiv (abs)PDF HTML

Papers citing "Quadratic Suffices for Over-parametrization via Matrix Chernoff Bound"

50 / 75 papers shown

Title
Data Uniformity Improves Training Efficiency and More, with a Convergence Framework Beyond the NTK Regime Yuqing Wang Shangding Gu 93 0 0 30 Jun 2025
Evaluating the design space of diffusion-based generative modelsNeural Information Processing Systems (NeurIPS), 2024 Yuqing Wang Ye He Molei Tao DiffM 215 14 0 18 Jun 2024
Approximation and Gradient Descent Training with Neural Networks G. Welper 113 2 0 19 May 2024
Analyzing the Neural Tangent Kernel of Periodically Activated Coordinate Networks Hemanth Saratchandran Shin-Fang Chng Simon Lucey 156 2 0 07 Feb 2024
A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative ModelsAnnual Review of Statistics and Its Application (ARSIA), 2024 Namjoon Suh Guang Cheng MedIm 208 17 0 14 Jan 2024
An Automatic Learning Rate Schedule Algorithm for Achieving Faster Convergence and Steeper Descent Zhao Song Chiwun Yang 148 10 0 17 Oct 2023
On the Convergence of Federated Averaging under Partial Participation for Over-parameterized Neural Networks Xin Liu Wei Tao Dazhi Zhan Yu Pan Xin Ma Yu Ding Zhisong Pan FedML 147 0 0 09 Oct 2023
How many Neurons do we need? A refined Analysis for Shallow Networks trained with Gradient DescentJournal of Statistical Planning and Inference (JSPI), 2023 Mike Nguyen Nicole Mücke MLT 149 6 0 14 Sep 2023
Approximation Results for Gradient Descent trained Neural Networks G. Welper 98 1 0 09 Sep 2023
Six Lectures on Linearized Neural NetworksJournal of Statistical Mechanics: Theory and Experiment (J. Stat. Mech.), 2023 Theodor Misiakiewicz Andrea Montanari 224 16 0 25 Aug 2023
How to Protect Copyright Data in Optimization of Large Language Models?AAAI Conference on Artificial Intelligence (AAAI), 2023 T. Chu Zhao Song Chiwun Yang 136 36 0 23 Aug 2023
Convergence of Two-Layer Regression with Nonlinear Units Yichuan Deng Zhao Song Shenghao Xie 143 8 0 16 Aug 2023
Memory capacity of two layer neural networks with smooth activationsSIAM Journal on Mathematics of Data Science (SIMODS), 2023 Liam Madden Christos Thrampoulidis MLT 212 6 0 03 Aug 2023
Controlling the Inductive Bias of Wide Neural Networks by Modifying the Kernel's Spectrum Amnon Geifman Daniel Barzilai Ronen Basri Meirav Galun 192 9 0 26 Jul 2023
Understanding Deep Neural Networks via Linear Separability of Hidden Layers Chao Zhang Xinyuan Chen Wensheng Li Lixue Liu Wei Wu Dacheng Tao 128 4 0 26 Jul 2023
Efficient SGD Neural Network Training via Sublinear Activated Neuron IdentificationBigData Congress [Services Society] (BSS), 2023 Lianke Qin Zhao Song Yuanyuan Yang 118 9 0 13 Jul 2023
Query Complexity of Active Learning for Function Family With Nearly Orthogonal Basis Xiangyi Chen Zhao Song Baochen Sun Junze Yin Danyang Zhuo 150 4 0 06 Jun 2023
Efficient Asynchronize Stochastic Gradient Algorithm with Structured Data Zhao Song Mingquan Ye 143 4 0 13 May 2023
An Over-parameterized Exponential Regression Yeqi Gao Sridhar Mahadevan Zhao Song 132 42 0 29 Mar 2023
Streaming Kernel PCA Algorithm With Small Space Yichuan Deng Zhao Song Zifan Wang Hangke Zhang 156 4 0 08 Mar 2023
Pruning Before Training May Improve Generalization, Provably Hongru Yang Yingbin Liang Xiaojie Guo Lingfei Wu Zhangyang Wang MLT 163 2 0 01 Jan 2023
Improved Convergence Guarantees for Shallow Neural Networks A. Razborov ODL 149 1 0 05 Dec 2022
Bypass Exponential Time Preprocessing: Fast Neural Network Training via Weight-Data Correlation PreprocessingNeural Information Processing Systems (NeurIPS), 2022 Josh Alman Jiehao Liang Zhao Song Ruizhe Zhang Danyang Zhuo 222 31 0 25 Nov 2022
Finite Sample Identification of Wide Shallow Neural Networks with Biases M. Fornasier T. Klock Marco Mondelli Michael Rauchensteiner 117 6 0 08 Nov 2022
Approximation results for Gradient Descent trained Shallow Neural Networks in $1d$ R. Gentile G. Welper ODL 158 9 0 17 Sep 2022
Neural Tangent Kernel: A Survey Eugene Golikov Eduard Pokonechnyy Vladimir Korviakov 150 17 0 29 Aug 2022
A Sublinear Adversarial Training AlgorithmInternational Conference on Learning Representations (ICLR), 2022 Yeqi Gao Lianke Qin Zhao Song Yitan Wang GAN 131 26 0 10 Aug 2022
Training Overparametrized Neural Networks in Sublinear Time Yichuan Deng Han Hu Zhao Song Omri Weinstein Danyang Zhuo BDL 157 28 0 09 Aug 2022
Provable Acceleration of Nesterov's Accelerated Gradient Method over Heavy Ball Method in Training Over-Parameterized Neural NetworksInternational Joint Conference on Artificial Intelligence (IJCAI), 2022 Xin Liu Wei Tao Wei Li Dazhi Zhan Jun Wang Zhisong Pan ODL 235 1 0 08 Aug 2022
Federated Adversarial Learning: A Framework with Convergence AnalysisInternational Conference on Machine Learning (ICML), 2022 Xiaoxiao Li Zhao Song Jiaming Yang FedML 158 28 0 07 Aug 2022
Bounding the Width of Neural Networks via Coupled Initialization -- A Worst Case AnalysisInternational Conference on Machine Learning (ICML), 2022 Alexander Munteanu Simon Omlor Zhao Song David P. Woodruff 138 16 0 26 Jun 2022
Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterizationNeural Information Processing Systems (NeurIPS), 2022 Simone Bombari Mohammad Hossein Amani Marco Mondelli 136 33 0 20 May 2022
Implicit Bias of MSE Gradient Optimization in Underparameterized Neural NetworksInternational Conference on Learning Representations (ICLR), 2022 Benjamin Bowman Guido Montúfar 142 13 0 12 Jan 2022
Training Multi-Layer Over-Parametrized Neural Network in Subquadratic Time Zhao Song Licheng Zhang Ruizhe Zhang 204 67 0 14 Dec 2021
On the Convergence of Shallow Neural Network Training with Randomly Masked Neurons Fangshuo Liao Anastasios Kyrillidis 217 16 0 05 Dec 2021
Fast Graph Neural Tangent Kernel via Kronecker SketchingAAAI Conference on Artificial Intelligence (AAAI), 2021 Shunhua Jiang Yunze Man Zhao Song Zheng Yu Danyang Zhuo 155 8 0 04 Dec 2021
Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models Tri Dao Beidi Chen Kaizhao Liang Jiaming Yang Zhao Song Atri Rudra Christopher Ré 261 87 0 30 Nov 2021
Subquadratic Overparameterization for Shallow Neural NetworksNeural Information Processing Systems (NeurIPS), 2021 Chaehwan Song Ali Ramezani-Kebrya Thomas Pethick Armin Eftekhari Volkan Cevher 135 33 0 02 Nov 2021
Does Preprocessing Help Training Over-parameterized Neural Networks?Neural Information Processing Systems (NeurIPS), 2021 Zhao Song Shuo Yang Ruizhe Zhang 170 50 0 09 Oct 2021
Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networks Zhichao Wang Yizhe Zhu 185 23 0 20 Sep 2021
Fast Sketching of Polynomial Kernels of Polynomial Degree Zhao Song David P. Woodruff Zheng Yu Lichen Zhang 146 46 0 21 Aug 2021
Provable Convergence of Nesterov's Accelerated Gradient Method for Over-Parameterized Neural Networks Xin Liu Zhisong Pan Wei Tao 215 9 0 05 Jul 2021
Early-stopped neural networks are consistentNeural Information Processing Systems (NeurIPS), 2021 Ziwei Ji Justin D. Li Matus Telgarsky 166 45 0 10 Jun 2021
FL-NTK: A Neural Tangent Kernel-based Framework for Federated Learning Convergence Analysis Baihe Huang Xiaoxiao Li Zhao Song Xin Yang FedML 107 16 0 11 May 2021
GIST: Distributed Training for Large-Scale Graph Convolutional NetworksJournal of Applied and Computational Topology (JACT), 2021 Cameron R. Wolfe Jingkang Yang Arindam Chowdhury Chen Dun Artun Bayer Santiago Segarra Anastasios Kyrillidis BDL GNN LRM 168 11 0 20 Feb 2021
On the Proof of Global Convergence of Gradient Descent for Deep ReLU Networks with Linear WidthsInternational Conference on Machine Learning (ICML), 2021 Quynh N. Nguyen 165 51 0 24 Jan 2021
A Convergence Theory Towards Practical Over-parameterized Deep Neural Networks Asaf Noy Yi Tian Xu Y. Aflalo Lihi Zelnik-Manor Rong Jin 173 3 0 12 Jan 2021
Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU NetworksInternational Conference on Machine Learning (ICML), 2020 Quynh N. Nguyen Marco Mondelli Guido Montúfar 328 92 0 21 Dec 2020
Metric Transforms and Low Rank Matrices via Representation Theory of the Real Hyperrectangle Josh Alman T. Chu Gary Miller Shyam Narayanan Mark Sellke Zhao Song 90 1 0 23 Nov 2020
Algorithms and Hardness for Linear Algebra on Geometric Graphs Josh Alman T. Chu Aaron Schild Zhao Song 200 31 0 04 Nov 2020