$u-$\mu$P: The Unit-Scaled Maximal Update Parametrization$

u- $\mu$ P: The Unit-Scaled Maximal Update Parametrization

24 July 2024

Bjorn Deiseroth

Andres Felipe Cruz Salinas

Carlo Luschi

Samuel Weinbach

Papers citing "u-$\mu$P: The Unit-Scaled Maximal Update Parametrization"

9 / 9 papers shown

Title
Don't be lazy: CompleteP enables compute-efficient deep transformers Nolan Dey Bin Claire Zhang Lorenzo Noci Mufan Bill Li Blake Bordelon Shane Bergsma C. Pehlevan Boris Hanin Joel Hestness 35 0 0 02 May 2025
Time Transfer: On Optimal Learning Rate and Batch Size In The Infinite Data Limit Oleg Filatov Jan Ebert Jiangtao Wang Stefan Kesselheim 29 3 0 10 Jan 2025
Towards Precise Scaling Laws for Video Diffusion Transformers Yuanyang Yin Yaqi Zhao Mingwu Zheng Ke Lin Jiarong Ou ... Pengfei Wan Di Zhang Baoqun Yin Wentao Zhang Kun Gai 110 2 0 03 Jan 2025
A Hitchhiker's Guide to Scaling Law Estimation Leshem Choshen Yang Zhang Jacob Andreas 28 6 0 15 Oct 2024
Scaling Optimal LR Across Token Horizons Johan Bjorck Alon Benhaim Vishrav Chaudhary Furu Wei Xia Song 38 4 0 30 Sep 2024
Massive Activations in Large Language Models Mingjie Sun Xinlei Chen J. Zico Kolter Zhuang Liu 52 64 0 27 Feb 2024
The Falcon Series of Open Language Models Ebtesam Almazrouei Hamza Alobeidli Abdulaziz Alshamsi Alessandro Cappelli Ruxandra-Aimée Cojocaru ... Quentin Malartic Daniele Mazzotta Badreddine Noune B. Pannier Guilherme Penedo AI4TS ALM 104 389 0 28 Nov 2023
FP8-LM: Training FP8 Large Language Models Houwen Peng Kan Wu Yixuan Wei Guoshuai Zhao Yuxiang Yang ... Zheng-Wei Zhang Shuguang Liu Joe Chau Han Hu Peng Cheng MQ 54 37 0 27 Oct 2023
FP8 Formats for Deep Learning Paulius Micikevicius Dusan Stosic N. Burgess Marius Cornea Pradeep Dubey ... Naveen Mellempudi S. Oberman M. Shoeybi Michael Siu Hao Wu BDL VLM MQ 62 119 0 12 Sep 2022