Unit Scaling: Out-of-the-Box Low-Precision Training

Unit Scaling: Out-of-the-Box Low-Precision Training

20 March 2023

Carlo Luschi

Papers citing "Unit Scaling: Out-of-the-Box Low-Precision Training"

10 / 10 papers shown

Title
$u-$\mu$P: The Unit-Scaled Maximal Update Parametrization$ u- $\mu$ P: The Unit-Scaled Maximal Update Parametrization Charlie Blake C. Eichenberg Josef Dean Lukas Balles Luke Y. Prince Bjorn Deiseroth Andres Felipe Cruz Salinas Carlo Luschi Samuel Weinbach Douglas Orr 51 9 0 24 Jul 2024
Scalify: scale propagation for efficient low-precision LLM training Paul Balança Sam Hosegood Carlo Luschi Andrew Fitzgibbon 24 2 0 24 Jul 2024
eXmY: A Data Type and Technique for Arbitrary Bit Precision Quantization Aditya Agrawal Matthew Hedlund Blake A. Hechtman MQ 13 4 0 22 May 2024
Training and inference of large language models using 8-bit floating point Sergio P. Perez Yan Zhang James Briggs Charlie Blake P. Krishnamurthy Paul Balanca Carlo Luschi Stephen Barlow Andrew William Fitzgibbon MQ 19 18 0 29 Sep 2023
Stable and low-precision training for large-scale vision-language models Mitchell Wortsman Tim Dettmers Luke Zettlemoyer Ari S. Morcos Ali Farhadi Ludwig Schmidt MQ MLLM VLM 16 38 0 25 Apr 2023
FP8 Formats for Deep Learning Paulius Micikevicius Dusan Stosic N. Burgess Marius Cornea Pradeep Dubey ... Naveen Mellempudi S. Oberman M. Shoeybi Michael Siu Hao Wu BDL VLM MQ 67 121 0 12 Sep 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 303 11,881 0 04 Mar 2022
High-Performance Large-Scale Image Recognition Without Normalization Andrew Brock Soham De Samuel L. Smith Karen Simonyan VLM 220 510 0 11 Feb 2021
Extracting Training Data from Large Language Models Nicholas Carlini Florian Tramèr Eric Wallace Matthew Jagielski Ariel Herbert-Voss ... Tom B. Brown D. Song Ulfar Erlingsson Alina Oprea Colin Raffel MLAU SILM 267 1,808 0 14 Dec 2020
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation Yonghui Wu M. Schuster Z. Chen Quoc V. Le Mohammad Norouzi ... Alex Rudnick Oriol Vinyals G. Corrado Macduff Hughes J. Dean AIMat 716 6,724 0 26 Sep 2016