Practical tradeoffs between memory, compute, and performance in learned optimizers

22 March 2022

Jascha Narain Sohl-Dickstein

ArXiv PDF HTML

Papers citing "Practical tradeoffs between memory, compute, and performance in learned optimizers"

27 / 27 papers shown

Title
Scalable Meta-Learning via Mixed-Mode Differentiation Iurii Kemaev Dan A Calian Luisa M Zintgraf Gregory Farquhar H. V. Hasselt 54 0 0 01 May 2025
Learning Versatile Optimizers on a Compute Diet A. Moudgil Boris Knyazev Guillaume Lajoie Eugene Belilovsky 93 0 0 22 Jan 2025
A Generalization Result for Convergence in Learning-to-Optimize Michael Sucker Peter Ochs 26 0 0 10 Oct 2024
Learning to Explore for Stochastic Gradient MCMC Seunghyun Kim Seohyeon Jung Seonghyeon Kim Juho Lee BDL 30 1 0 17 Aug 2024
Can Learned Optimization Make Reinforcement Learning Less Difficult? Alexander David Goldie Chris Xiaoxuan Lu Matthew Jackson Shimon Whiteson Jakob N. Foerster 40 3 0 09 Jul 2024
$μ$ LO: Compute-Efficient Meta-Generalization of Learned Optimizers Benjamin Thérien Charles-Étienne Joseph Boris Knyazev Edouard Oyallon Irina Rish Eugene Belilovsky AI4CE 32 1 0 31 May 2024
From Learning to Optimize to Learning Optimization Algorithms Camille Castera Peter Ochs 57 1 0 28 May 2024
Graph Neural Networks for Learning Equivariant Representations of Neural Networks Miltiadis Kofinas Boris Knyazev Yan Zhang Yunlu Chen Gertjan J. Burghouts E. Gavves Cees G. M. Snoek David W. Zhang 44 29 0 18 Mar 2024
QN-Mixer: A Quasi-Newton MLP-Mixer Model for Sparse-View CT Reconstruction Ishak Ayad Nicolas Larue Mai K. Nguyen 33 3 0 28 Feb 2024
Dynamic Memory Based Adaptive Optimization Balázs Szegedy Domonkos Czifra Péter Korösi-Szabó ODL 27 0 0 23 Feb 2024
Can We Learn Communication-Efficient Optimizers? Charles-Étienne Joseph Benjamin Thérien A. Moudgil Boris Knyazev Eugene Belilovsky 26 1 0 02 Dec 2023
Towards Constituting Mathematical Structures for Learning to Optimize Jialin Liu Xiaohan Chen Zhangyang Wang W. Yin HanQin Cai 21 12 0 29 May 2023
Variance-Reduced Gradient Estimation via Noise-Reuse in Online Evolution Strategies Oscar Li James Harrison Jascha Narain Sohl-Dickstein Virginia Smith Luke Metz 39 5 0 21 Apr 2023
Improving physics-informed neural networks with meta-learned optimization Alexander Bihlo PINN 26 18 0 13 Mar 2023
Learning to Optimize for Reinforcement Learning Qingfeng Lan Rupam Mahmood Shuicheng Yan Zhongwen Xu OffRL 24 6 0 03 Feb 2023
A Nonstochastic Control Approach to Optimization Xinyi Chen Elad Hazan 40 5 0 19 Jan 2023
evosax: JAX-based Evolution Strategies R. T. Lange 28 54 0 08 Dec 2022
Transformer-Based Learned Optimization Erik Gartner Luke Metz Mykhaylo Andriluka C. Freeman C. Sminchisescu 16 11 0 02 Dec 2022
Discovering Evolution Strategies via Meta-Black-Box Optimization R. T. Lange Tom Schaul Yutian Chen Tom Zahavy Valenti Dallibard Chris Xiaoxuan Lu Satinder Singh Sebastian Flennerhag 33 47 0 21 Nov 2022
VeLO: Training Versatile Learned Optimizers by Scaling Up Luke Metz James Harrison C. Freeman Amil Merchant Lucas Beyer ... Naman Agrawal Ben Poole Igor Mordatch Adam Roberts Jascha Narain Sohl-Dickstein 21 60 0 17 Nov 2022
Discovered Policy Optimisation Chris Xiaoxuan Lu J. Kuba Alistair Letcher Luke Metz Christian Schroeder de Witt Jakob N. Foerster OffRL 23 74 0 11 Oct 2022
Learning to Learn with Generative Models of Neural Network Checkpoints William S. Peebles Ilija Radosavovic Tim Brooks Alexei A. Efros Jitendra Malik UQCV 73 64 0 26 Sep 2022
A Closer Look at Learned Optimization: Stability, Robustness, and Inductive Biases James Harrison Luke Metz Jascha Narain Sohl-Dickstein 42 22 0 22 Sep 2022
Symbolic Learning to Optimize: Towards Interpretability and Scalability Wenqing Zheng Tianlong Chen Ting-Kuei Hu Zhangyang Wang 37 18 0 13 Mar 2022
Learning to Optimize: A Primer and A Benchmark Tianlong Chen Xiaohan Chen Wuyang Chen Howard Heaton Jialin Liu Zhangyang Wang W. Yin 23 223 0 23 Mar 2021
Training Learned Optimizers with Randomly Initialized Learned Optimizers Luke Metz C. Freeman Niru Maheswaranathan Jascha Narain Sohl-Dickstein 41 12 0 14 Jan 2021
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks Chelsea Finn Pieter Abbeel Sergey Levine OOD 243 11,659 0 09 Mar 2017