Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves

23 September 2020

Jascha Narain Sohl-Dickstein

ArXiv PDF HTML

Papers citing "Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves"

50 / 52 papers shown

Title
Phoeni6: a Systematic Approach for Evaluating the Energy Consumption of Neural Networks Antônio Oliveira-Filho Wellington Silva-de-Souza Carlos Alberto Valderrama Sakuyama Samuel Xavier-de-Souza 63 0 0 25 Feb 2025
Learning Versatile Optimizers on a Compute Diet A. Moudgil Boris Knyazev Guillaume Lajoie Eugene Belilovsky 126 0 0 22 Jan 2025
Meta-Sparsity: Learning Optimal Sparse Structures in Multi-task Networks through Meta-learning Richa Upadhyay Ronald Phlypo Rajkumar Saini Marcus Liwicki 35 0 0 21 Jan 2025
Data Selection via Optimal Control for Language Models Yuxian Gu Li Dong Hongning Wang Y. Hao Qingxiu Dong Furu Wei Minlie Huang AI4CE 50 4 0 09 Oct 2024
Narrowing the Focus: Learned Optimizers for Pretrained Models Gus Kristiansen Mark Sandler A. Zhmoginov Nolan Miller Anirudh Goyal Jihwan Lee Max Vladymyrov 34 1 0 17 Aug 2024
Can Learned Optimization Make Reinforcement Learning Less Difficult? Alexander David Goldie Chris Xiaoxuan Lu Matthew Jackson Shimon Whiteson Jakob N. Foerster 40 3 0 09 Jul 2024
Implicit Neural Image Field for Biological Microscopy Image Compression Gaole Dai Cheng-Ching Tseng Qingpo Wuwu Rongyu Zhang Shaokang Wang ... Yu Zhou A. A. Tuz Matthias Gunzer Jianxu Chen Shanghang Zhang 25 1 0 29 May 2024
Graph Neural Networks for Learning Equivariant Representations of Neural Networks Miltiadis Kofinas Boris Knyazev Yan Zhang Yunlu Chen Gertjan J. Burghouts E. Gavves Cees G. M. Snoek David W. Zhang 44 29 0 18 Mar 2024
Dynamic Memory Based Adaptive Optimization Balázs Szegedy Domonkos Czifra Péter Korösi-Szabó ODL 27 0 0 23 Feb 2024
Moco: A Learnable Meta Optimizer for Combinatorial Optimization Tim Dernedde Daniela Thyssens Soren Dittrich Maximilan Stubbemann Lars Schmidt-Thieme 49 5 0 07 Feb 2024
MADA: Meta-Adaptive Optimizers through hyper-gradient Descent Kaan Ozkara Can Karakus Parameswaran Raman Mingyi Hong Shoham Sabach B. Kveton V. Cevher 19 2 0 17 Jan 2024
Investigation into the Training Dynamics of Learned Optimizers Jan Sobotka Petr Simánek Daniel Vasata 26 0 0 12 Dec 2023
Can We Learn Communication-Efficient Optimizers? Charles-Étienne Joseph Benjamin Thérien A. Moudgil Boris Knyazev Eugene Belilovsky 26 1 0 02 Dec 2023
Generalisable Agents for Neural Network Optimisation Kale-ab Tessera C. Tilbury Sasha Abramowitz Ruan de Kock Omayma Mahjoub Benjamin Rosman Sara Hooker Arnu Pretorius AI4CE 20 0 0 30 Nov 2023
Prompt Engineering a Prompt Engineer Qinyuan Ye Maxamed Axmed Reid Pryzant Fereshte Khani VLM LLMAG LRM 27 28 0 09 Nov 2023
Deep Model Predictive Optimization Jacob Sacks Rwik Rana Kevin Huang Alex Spitzer Guanya Shi Byron Boots 32 7 0 06 Oct 2023
Advances and Challenges in Meta-Learning: A Technical Review Anna Vettoruzzo Mohamed-Rafik Bouguelia Joaquin Vanschoren Thorsteinn Rögnvaldsson K. Santosh OffRL 19 70 0 10 Jul 2023
Towards Constituting Mathematical Structures for Learning to Optimize Jialin Liu Xiaohan Chen Zhangyang Wang W. Yin HanQin Cai 26 12 0 29 May 2023
HUB: Guiding Learned Optimizers with Continuous Prompt Tuning Gaole Dai Wei Yu Wu Ziyu Wang Jie Fu Shanghang Zhang Tiejun Huang AIFin 14 0 0 26 May 2023
Low-Variance Gradient Estimation in Unrolled Computation Graphs with ES-Single Paul Vicol Zico Kolter Kevin Swersky 13 6 0 21 Apr 2023
Discovering Attention-Based Genetic Algorithms via Meta-Black-Box Optimization R. T. Lange Tom Schaul Yutian Chen Chris Xiaoxuan Lu Tom Zahavy Valentin Dalibard Sebastian Flennerhag 24 34 0 08 Apr 2023
Learning to Optimize for Reinforcement Learning Qingfeng Lan Rupam Mahmood Shuicheng Yan Zhongwen Xu OffRL 24 6 0 03 Feb 2023
Mnemosyne: Learning to Train Transformers with Transformers Deepali Jain K. Choromanski Kumar Avinava Dubey Sumeet Singh Vikas Sindhwani Tingnan Zhang Jie Tan OffRL 31 9 0 02 Feb 2023
General-Purpose In-Context Learning by Meta-Learning Transformers Louis Kirsch James Harrison Jascha Narain Sohl-Dickstein Luke Metz 29 72 0 08 Dec 2022
Transformer-Based Learned Optimization Erik Gartner Luke Metz Mykhaylo Andriluka C. Freeman C. Sminchisescu 16 11 0 02 Dec 2022
Learning to Optimize with Dynamic Mode Decomposition Petr Simánek Daniel Vasata Pavel Kordík 31 5 0 29 Nov 2022
A Recursively Recurrent Neural Network (R2N2) Architecture for Learning Iterative Algorithms Danimir T. Doncevic Alexander Mitsos Yu Guo Qianxiao Li Felix Dietrich Manuel Dahmen Ioannis G. Kevrekidis 11 7 0 22 Nov 2022
Discovering Evolution Strategies via Meta-Black-Box Optimization R. T. Lange Tom Schaul Yutian Chen Tom Zahavy Valenti Dallibard Chris Xiaoxuan Lu Satinder Singh Sebastian Flennerhag 36 47 0 21 Nov 2022
VeLO: Training Versatile Learned Optimizers by Scaling Up Luke Metz James Harrison C. Freeman Amil Merchant Lucas Beyer ... Naman Agrawal Ben Poole Igor Mordatch Adam Roberts Jascha Narain Sohl-Dickstein 24 60 0 17 Nov 2022
Empirical Study on Optimizer Selection for Out-of-Distribution Generalization Hiroki Naganuma Kartik Ahuja S. Takagi Tetsuya Motokawa Rio Yokota Kohta Ishikawa I. Sato Ioannis Mitliagkas OOD 11 7 0 15 Nov 2022
Learning to Optimize Quasi-Newton Methods Isaac Liao Rumen Dangovski Jakob N. Foerster Marin Soljacic 36 4 0 11 Oct 2022
A Closer Look at Learned Optimization: Stability, Robustness, and Inductive Biases James Harrison Luke Metz Jascha Narain Sohl-Dickstein 44 22 0 22 Sep 2022
Automated Dynamic Algorithm Configuration Steven Adriaensen André Biedenkapp Gresa Shala Noor H. Awad Theresa Eimer Marius Lindauer Frank Hutter 27 36 0 27 May 2022
Practical tradeoffs between memory, compute, and performance in learned optimizers Luke Metz C. Freeman James Harrison Niru Maheswaranathan Jascha Narain Sohl-Dickstein 33 32 0 22 Mar 2022
A Simple Guard for Learned Optimizers Isabeau Prémont-Schwarz Jaroslav Vítkru Jan Feyereisl 43 7 0 28 Jan 2022
Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategies Paul Vicol Luke Metz Jascha Narain Sohl-Dickstein 14 67 0 27 Dec 2021
Efficient Meta Subspace Optimization Yoni Choukroun Michael Katz 15 1 0 28 Oct 2021
Parameter Prediction for Unseen Deep Architectures Boris Knyazev M. Drozdzal Graham W. Taylor Adriana Romero Soriano OOD 22 78 0 25 Oct 2021
Learn2Hop: Learned Optimization on Rough Landscapes Amil Merchant Luke Metz S. Schoenholz E. D. Cubuk 15 16 0 20 Jul 2021
Memory Augmented Optimizers for Deep Learning Paul-Aymeric McRae Prasanna Parthasarathi Mahmoud Assran Sarath Chandar ODL 17 3 0 20 Jun 2021
A Generalizable Approach to Learning Optimizers Diogo Almeida Clemens Winter Jie Tang Wojciech Zaremba AI4CE 19 29 0 02 Jun 2021
Meta-Learning Bidirectional Update Rules Mark Sandler Max Vladymyrov A. Zhmoginov Nolan Miller Andrew Jackson T. Madams Blaise Agüera y Arcas 11 15 0 10 Apr 2021
Learning to Optimize: A Primer and A Benchmark Tianlong Chen Xiaohan Chen Wuyang Chen Howard Heaton Jialin Liu Zhangyang Wang W. Yin 30 225 0 23 Mar 2021
Meta Learning Black-Box Population-Based Optimizers H. Gomes B. Léger Christian Gagné 11 12 0 05 Mar 2021
Training Learned Optimizers with Randomly Initialized Learned Optimizers Luke Metz C. Freeman Niru Maheswaranathan Jascha Narain Sohl-Dickstein 41 12 0 14 Jan 2021
Meta Learning Backpropagation And Improving It Louis Kirsch Jürgen Schmidhuber 40 56 0 29 Dec 2020
Reverse engineering learned optimizers reveals known and novel mechanisms Niru Maheswaranathan David Sussillo Luke Metz Ruoxi Sun Jascha Narain Sohl-Dickstein 6 21 0 04 Nov 2020
Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning Matthias Feurer Katharina Eggensperger Stefan Falkner Marius Lindauer Frank Hutter 33 266 0 08 Jul 2020
Meta-Learning in Neural Networks: A Survey Timothy M. Hospedales Antreas Antoniou P. Micaelli Amos Storkey OOD 38 1,926 0 11 Apr 2020
Bilevel Programming for Hyperparameter Optimization and Meta-Learning Luca Franceschi P. Frasconi Saverio Salzo Riccardo Grazzi Massimiliano Pontil 99 716 0 13 Jun 2018