Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer

7 March 2022

Xiaodong Liu

Papers citing "Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer"

50 / 122 papers shown

Title
Ultra-FineWeb: Efficient Data Filtering and Verification for High-Quality LLM Training Data Y. Wang Z. Fu Jie Cai Peijun Tang Hongya Lyu ... Jie Zhou Guoyang Zeng Chaojun Xiao Xu Han Zhiyuan Liu 47 0 0 08 May 2025
On Model and Data Scaling for Skeleton-based Self-Supervised Gait Recognition Adrian Cosma Andy Catruna Emilian Radoi 31 0 0 10 Apr 2025
Prot42: a Novel Family of Protein Language Models for Target-aware Protein Binder Generation Mohammad Amaan Sayeed Engin Tekin Maryam Nadeem Nancy A. ElNaker A. Singh Natalia Vassilieva Boulbaba Ben Amor 21 0 0 06 Apr 2025
Chem42: a Family of chemical Language Models for Target-aware Ligand Generation A. Singh Engin Tekin Maryam Nadeem Nancy A. ElNaker Mohammad Amaan Sayeed Natalia Vassilieva Boulbaba Ben Amor 45 1 0 20 Mar 2025
Flow to the Mode: Mode-Seeking Diffusion Autoencoders for State-of-the-Art Image Tokenization Kyle Sargent Kyle Hsu Justin Johnson L. Fei-Fei Jiajun Wu DiffM MU 53 2 0 14 Mar 2025
Learning richness modulates equality reasoning in neural networks William L. Tong C. Pehlevan 42 0 0 12 Mar 2025
(Mis)Fitting: A Survey of Scaling Laws Margaret Li Sneha Kudugunta Luke Zettlemoyer 69 2 0 26 Feb 2025
Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biases Michael Y. Hu Jackson Petty Chuan Shi William Merrill Tal Linzen AI4CE 62 1 0 26 Feb 2025
The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training Jinbo Wang Mingze Wang Zhanpeng Zhou Junchi Yan Weinan E Lei Wu 73 1 0 26 Feb 2025
Unveiling Downstream Performance Scaling of LLMs: A Clustering-Based Perspective Chengyin Xu Kaiyuan Chen Xiao Li Ke Shen Chenggang Li OffRL 41 0 0 24 Feb 2025
VaViM and VaVAM: Autonomous Driving through Video Generative Modeling Florent Bartoccioni Elias Ramzi Victor Besnier Shashanka Venkataramanan Tuan-Hung Vu ... Mickael Chen Éloi Zablocki Andrei Bursuc Eduardo Valle Matthieu Cord VGen 78 1 0 24 Feb 2025
Towards Precise Scaling Laws for Video Diffusion Transformers Yuanyang Yin Yaqi Zhao Mingwu Zheng Ke Lin Jiarong Ou ... Pengfei Wan Di Zhang Baoqun Yin Wentao Zhang Kun Gai 119 2 0 03 Jan 2025
Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs Aldo Pareja Nikhil Shivakumar Nayak Hao Wang Krishnateja Killamsetty Shivchander Sudalairaj ... Guangxuan Xu Kai Xu Ligong Han Luke Inglis Akash Srivastava 78 6 0 17 Dec 2024
Model Fusion through Bayesian Optimization in Language Model Fine-Tuning Chaeyun Jang Hyungi Lee Jungtaek Kim Juho Lee MoMe 24 0 0 11 Nov 2024
Scaling Laws for Precision Tanishq Kumar Zachary Ankner Benjamin Spector Blake Bordelon Niklas Muennighoff Mansheej Paul C. Pehlevan Christopher Ré Aditi Raghunathan AIFin MoMe 46 12 0 07 Nov 2024
Crystal: Illuminating LLM Abilities on Language and Code Tianhua Tao Junbo Li Bowen Tan Hongyi Wang William Marshall ... Joel Hestness Natalia Vassilieva Zhiqiang Shen Eric P. Xing Zhengzhong Liu 40 4 0 06 Nov 2024
Sparsing Law: Towards Large Language Models with Greater Activation Sparsity Yuqi Luo Chenyang Song Xu Han Y. Chen Chaojun Xiao Zhiyuan Liu Maosong Sun 47 3 0 04 Nov 2024
How Does Critical Batch Size Scale in Pre-training? Hanlin Zhang Depen Morwani Nikhil Vyas Jingfeng Wu Difan Zou Udaya Ghai Dean Phillips Foster Sham Kakade 64 8 0 29 Oct 2024
A Random Matrix Theory Perspective on the Spectrum of Learned Features and Asymptotic Generalization Capabilities Yatin Dandi Luca Pesce Hugo Cui Florent Krzakala Yue M. Lu Bruno Loureiro MLT 35 1 0 24 Oct 2024
Estimating the Spectral Moments of the Kernel Integral Operator from Finite Sample Matrices Chanwoo Chun SueYeon Chung Daniel D. Lee 19 1 0 23 Oct 2024
A Hitchhiker's Guide to Scaling Law Estimation Leshem Choshen Yang Zhang Jacob Andreas 41 6 0 15 Oct 2024
Scaling Laws for Predicting Downstream Performance in LLMs Yangyi Chen Binxuan Huang Yifan Gao Zhengyang Wang Jingfeng Yang Heng Ji LRM 43 7 0 11 Oct 2024
Scaling Laws Across Model Architectures: A Comparative Analysis of Dense and MoE Models in Large Language Models Siqi Wang Zhengyu Chen Bei Li Keqing He Min Zhang Jingang Wang 23 2 0 08 Oct 2024
The Optimization Landscape of SGD Across the Feature Learning Strength Alexander B. Atanasov Alexandru Meterez James B. Simon C. Pehlevan 43 2 0 06 Oct 2024
Searching for Efficient Linear Layers over a Continuous Space of Structured Matrices Andres Potapczynski Shikai Qiu Marc Finzi Christopher Ferri Zixi Chen Micah Goldblum Bayan Bruss Christopher De Sa Andrew Gordon Wilson 24 1 0 03 Oct 2024
Theoretical Insights into Fine-Tuning Attention Mechanism: Generalization and Optimization Xinhao Yao Hongjin Qian Xiaolin Hu Gengze Xu Wei Liu Jian Luan B. Wang Y. Liu 45 0 0 03 Oct 2024
Scaling Optimal LR Across Token Horizons Johan Bjorck Alon Benhaim Vishrav Chaudhary Furu Wei Xia Song 46 4 0 30 Sep 2024
How Feature Learning Can Improve Neural Scaling Laws Blake Bordelon Alexander B. Atanasov C. Pehlevan 46 12 0 26 Sep 2024
From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks Clémentine Dominé Nicolas Anguita A. Proca Lukas Braun D. Kunin P. Mediano Andrew M. Saxe 30 3 0 22 Sep 2024
Large-Scale Multi-omic Biosequence Transformers for Modeling Protein-Nucleic Acid Interactions Sully F. Chen Robert J. Steele Beakal Lemeneh S. Lad Eric Oermann Eric K. Oermann AI4CE 33 0 0 29 Aug 2024
Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Scheduler Yikang Shen Matthew Stallone Mayank Mishra Gaoyuan Zhang Shawn Tan Aditya Prasad Adriana Meza Soria David D. Cox Rameswar Panda 26 11 0 23 Aug 2024
Apple Intelligence Foundation Language Models Tom Gunter Zirui Wang Chong-Jun Wang Ruoming Pang Andy Narayanan ... Xinwen Liu Yang Zhao Yin Xia Zhile Ren Zhongzheng Ren 24 32 0 29 Jul 2024
Efficient LLM Training and Serving with Heterogeneous Context Sharding among Attention Heads Xihui Lin Yunan Zhang Suyu Ge Barun Patra Vishrav Chaudhary Hao Peng Xia Song 27 0 0 25 Jul 2024
$u-$\mu$P: The Unit-Scaled Maximal Update Parametrization$ u- $\mu$ P: The Unit-Scaled Maximal Update Parametrization Charlie Blake C. Eichenberg Josef Dean Lukas Balles Luke Y. Prince Bjorn Deiseroth Andres Felipe Cruz Salinas Carlo Luschi Samuel Weinbach Douglas Orr 51 9 0 24 Jul 2024
LoRA-GA: Low-Rank Adaptation with Gradient Approximation Shaowen Wang Linxi Yu Jian Li ALM AI4CE 26 26 0 06 Jul 2024
$$\text{Memory}^3$: Language Modeling with Explicit Memory$ $\text{Memory}^3$ : Language Modeling with Explicit Memory Hongkang Yang Zehao Lin Wenjin Wang Hao Wu Zhiyu Li ... Yu Yu Kai Chen Feiyu Xiong Linpeng Tang Weinan E 35 11 0 01 Jul 2024
Hyperparameter Optimization for Randomized Algorithms: A Case Study on Random Features Oliver R. A. Dunbar Nicholas H. Nelsen Maya Mutic 18 5 0 30 Jun 2024
The Impact of Initialization on LoRA Finetuning Dynamics Soufiane Hayou Nikhil Ghosh Bin Yu AI4CE 34 10 0 12 Jun 2024
Compute Better Spent: Replacing Dense Layers with Structured Matrices Shikai Qiu Andres Potapczynski Marc Finzi Micah Goldblum Andrew Gordon Wilson 27 11 0 10 Jun 2024
Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning D. Kunin Allan Raventós Clémentine Dominé Feng Chen David Klindt Andrew M. Saxe Surya Ganguli MLT 35 15 0 10 Jun 2024
$μ$ LO: Compute-Efficient Meta-Generalization of Learned Optimizers Benjamin Thérien Charles-Étienne Joseph Boris Knyazev Edouard Oyallon Irina Rish Eugene Belilovsky AI4CE 25 1 0 31 May 2024
Recurrent neural networks: vanishing and exploding gradients are not the end of the story Nicolas Zucchet Antonio Orvieto ODL AAML 40 8 0 31 May 2024
How to set AdamW's weight decay as you scale model and dataset size Xi Wang Laurence Aitchison 38 9 0 22 May 2024
LaT-PFN: A Joint Embedding Predictive Architecture for In-context Time-series Forecasting Stijn Verdenius Andrea Zerio Roy L.M. Wang BDL AI4TS AI4CE 27 2 0 16 May 2024
Towards a Theoretical Understanding of the 'Reversal Curse' via Training Dynamics Hanlin Zhu Baihe Huang Shaolun Zhang Michael I. Jordan Jiantao Jiao Yuandong Tian Stuart Russell LRM AI4CE 44 13 0 07 May 2024
Temporal Scaling Law for Large Language Models Yizhe Xiong Xiansheng Chen Xin Ye Hui Chen Zijia Lin ... Zhenpeng Su Jianwei Niu Guiguang Ding Jungong Han Guiguang Ding 36 9 0 27 Apr 2024
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone Marah Abdin Sam Ade Jacobs A. A. Awan J. Aneja Ahmed Hassan Awadallah ... Li Lyna Zhang Yi Zhang Yue Zhang Yunan Zhang Xiren Zhou LRM ALM 50 995 0 22 Apr 2024
MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies Shengding Hu Yuge Tu Xu Han Chaoqun He Ganqu Cui ... Chaochao Jia Guoyang Zeng Dahai Li Zhiyuan Liu Maosong Sun MoE 38 275 0 09 Apr 2024
MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training Brandon McKinzie Zhe Gan J. Fauconnier Sam Dodge Bowen Zhang ... Zirui Wang Ruoming Pang Peter Grasch Alexander Toshev Yinfei Yang MLLM 27 185 0 14 Mar 2024
Simple and Scalable Strategies to Continually Pre-train Large Language Models Adam Ibrahim Benjamin Thérien Kshitij Gupta Mats L. Richter Quentin Anthony Timothée Lesort Eugene Belilovsky Irina Rish KELM CLL 44 50 0 13 Mar 2024