Training Deep Neural Networks with 8-bit Floating Point Numbers

19 December 2018

Papers citing "Training Deep Neural Networks with 8-bit Floating Point Numbers"

50 / 212 papers shown

Title
FP8-Flow-MoE: A Casting-Free FP8 Recipe without Double Quantization Error Fengjuan Wang Zhiyi Su Xingzhu Hu Cheng Wang Mou Sun MQ 44 0 0 04 Nov 2025
Training with Fewer Bits: Unlocking Edge LLMs Training with Stochastic RoundingConference on Empirical Methods in Natural Language Processing (EMNLP), 2025 Taowen Liu Marta Andronic Deniz Gündüz George A. Constantinides MQ 52 0 0 02 Nov 2025
FALQON: Accelerating LoRA Fine-tuning with Low-Bit Floating-Point Arithmetic Kanghyun Choi Hyeyoon Lee S. Park Dain Kwon Jinho Lee MQ 72 0 0 28 Oct 2025
InfiR2: A Comprehensive FP8 Training Recipe for Reasoning-Enhanced Language Models Wenjun Wang Shuo Cai C. Xie Mingfa Feng Y. Zhang Zhen Li Kejing Yang Ming Li Jiannong Cao Yuan Xie MQ 84 0 0 26 Sep 2025
Tri-Accel: Curvature-Aware Precision-Adaptive and Memory-Elastic Optimization for Efficient GPU Usage Mohsen Sheibanian Pouya Shaeri Alimohammad Beigi Ryan T. Woo Aryan Keluskar 95 0 0 23 Aug 2025
REG4Rec: Reasoning-Enhanced Generative Model for Large-Scale Recommendation Systems Haibo Xing Hao Deng Yucheng Mao Jinxin Hu Y. Xu ... Jiahao Wang Shizhun Wang Yu Zhang Xiaoyi Zeng Jing-Xuan Zhang LRM 90 1 0 21 Aug 2025
Compression-Induced Communication-Efficient Large Model Training and Inferencing Sudip K. Seal Maksudul Alam Jorge Ramirez Sajal Dash Hao Lu AI4CE 46 0 0 01 Aug 2025
Pimba: A Processing-in-Memory Acceleration for Post-Transformer Large Language Model Serving Wonung Kim Yubin Lee Yoonsung Kim Jinwoo Hwang Seongryong Oh ... Aziz Huseynov Woong Gyu Park Chang Hyun Park Divya Mahajan Jongse Park 437 1 0 14 Jul 2025
Beyond Discreteness: Finite-Sample Analysis of Straight-Through Estimator for Quantization Halyun Jeong Jack Xin Penghang Yin MQ 131 0 0 23 May 2025
Pushing the Limits of Low-Bit Optimizers: A Focus on EMA Dynamics Cong Xu Wenbin Liang Mo Yu Anan Liu Jianchao Tan Lizhuang Ma Jiangming Wang Jun Wang Weinan Zhang Wei Zhang MQ 233 0 0 01 May 2025
Morphing-based Compression for Data-centric ML Pipelines Sebastian Baunsgaard Matthias Boehm 129 0 0 15 Apr 2025
Mixed precision accumulation for neural network inference guided by componentwise forward error analysis El-Mehdi El Arar Silviu-Ioan Filip Theo Mary Elisa Riccietti 157 0 0 19 Mar 2025
Stochastic Rounding for LLM Training: Theory and PracticeInternational Conference on Artificial Intelligence and Statistics (AISTATS), 2025 Kaan Ozkara Tao Yu Youngsuk Park 129 2 0 27 Feb 2025
Stable-SPAM: How to Train in 4-Bit More Stably than 16-Bit Adam Tianjin Huang Haotian Hu Zhenyu Zhang Gaojie Jin Xianrui Li ... Tianlong Chen Lu Liu Qingsong Wen Zhangyang Wang Shiwei Liu MQ 265 5 0 24 Feb 2025
GSQ-Tuning: Group-Shared Exponents Integer in Fully Quantized Training for LLMs On-Device Fine-tuningAnnual Meeting of the Association for Computational Linguistics (ACL), 2025 Sifan Zhou Shuo Wang Zhihang Yuan Mingjia Shi Yuzhang Shang Dawei Yang MQ ALM 420 8 0 18 Feb 2025
QuZO: Quantized Zeroth-Order Fine-Tuning for Large Language Models Jiajun Zhou Yifan Yang Kai Zhen Ziyue Liu Yequan Zhao Ershad Banijamali Athanasios Mouchtaris Ngai Wong Zheng Zhang MQ 208 4 0 17 Feb 2025
Optimizing Large Language Model Training Using FP4 Quantization Ruizhe Wang Yeyun Gong Xiao Liu Guoshuai Zhao Ziyue Yang Baining Guo Zhengjun Zha Peng Cheng MQ 290 28 0 28 Jan 2025
Deterministic and Probabilistic Rounding Error Analysis for Mixed-Precision Arithmetic on Modern Computing Units Sahil Bhola Karthik Duraisamy 113 2 0 27 Nov 2024
AMXFP4: Taming Activation Outliers with Asymmetric Microscaling Floating-Point for 4-bit LLM InferenceAnnual Meeting of the Association for Computational Linguistics (ACL), 2024 Janghwan Lee Jiwoong Park Jinseok Kim Yongjik Kim Jungju Oh Jinwook Oh Jungwook Choi 203 8 0 15 Nov 2024
COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 TrainingInternational Conference on Learning Representations (ICLR), 2024 Haocheng Xi Han Cai Ligeng Zhu Yaojie Lu Kurt Keutzer Jianfei Chen Song Han MQ 327 14 0 25 Oct 2024
Understanding Adam Requires Better Rotation Dependent Assumptions Tianyue H. Zhang Lucas Maes Alexia Jolicoeur-Martineau Alexia Jolicoeur-Martineau Damien Scieur Damien Scieur Simon Lacoste-Julien Charles Guille-Escuret 170 6 0 25 Oct 2024
FusionLLM: A Decentralized LLM Training System on Geo-distributed GPUs with Adaptive Compression Zhenheng Tang Xueze Kang Yiming Yin Xinglin Pan Yuxin Wang ... Shaohuai Shi Amelie Chi Zhou Bo Li Bingsheng He Xiaowen Chu AI4CE 166 10 0 16 Oct 2024
Ascend HiFloat8 Format for Deep Learning Yuanyong Luo Zhongxing Zhang Richard Wu Hu Liu Ying Jin ... Korviakov Vladimir Bobrin Maxim Yuhao Hu Guanfu Chen Zeyi Huang MQ 88 2 0 25 Sep 2024
S-STE: Continuous Pruning Function for Efficient 2:4 Sparse Pre-trainingNeural Information Processing Systems (NeurIPS), 2024 Yuezhou Hu Jun-Jie Zhu Jianfei Chen 311 5 0 13 Sep 2024
1-Bit FQT: Pushing the Limit of Fully Quantized Training to 1-bit Chang Gao Jianfei Chen Kang Zhao Jiaqi Wang Liping Jing MQ 181 3 0 26 Aug 2024
Inference Optimizations for Large Language Models: Effects, Challenges, and Practical Considerations Leo Donisch Sigurd Schacht Carsten Lanquillon 166 3 0 06 Aug 2024
A Metric Driven Approach to Mixed Precision Training M. Rasquinha Gil Tabak 80 0 0 06 Aug 2024
Efficient Training of Large Language Models on Distributed Infrastructures: A Survey Jiangfei Duan Shuo Zhang Zerui Wang Lijuan Jiang Wenwen Qu ... Dahua Lin Yonggang Wen Xin Jin Tianwei Zhang Yang Liu 250 24 0 29 Jul 2024
$u-$\mu$P: The Unit-Scaled Maximal Update Parametrization$ u- $\mu$ P: The Unit-Scaled Maximal Update Parametrization Charlie Blake C. Eichenberg Josef Dean Lukas Balles Luke Y. Prince Bjorn Deiseroth Andres Felipe Cruz Salinas Carlo Luschi Samuel Weinbach Douglas Orr 188 16 0 24 Jul 2024
Scalify: scale propagation for efficient low-precision LLM training Paul Balança Sam Hosegood Carlo Luschi Andrew Fitzgibbon 115 4 0 24 Jul 2024
Inverted Activations Georgii Sergeevich Novikov Ivan Oseledets 89 0 0 22 Jul 2024
Toward INT4 Fixed-Point Training via Exploring Quantization Error for Gradients Jeimin Jeon Junghyup Lee Jeimin Jeon Jaehyeon Moon Bumsub Ham MQ 171 1 0 17 Jul 2024
On-Device Training of Fully Quantized Deep Neural Networks on Cortex-M Microcontrollers M. Deutel Frank Hannig Christopher Mutschler Jürgen Teich MQ 174 2 0 15 Jul 2024
On Exact Bit-level Reversible Transformers Without Changing Architectures Guoqiang Zhang J. P. Lewis W. Kleijn MQ AI4CE 109 1 0 12 Jul 2024
Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients Zhenyu Zhang Ajay Jaiswal L. Yin Shiwei Liu Jiawei Zhao Yuandong Tian Zhangyang Wang VLM 153 31 0 11 Jul 2024
SHERL: Synthesizing High Accuracy and Efficient Memory for Resource-Limited Transfer Learning Haiwen Diao Bo Wan Xu Jia Yunzhi Zhuge Ying Zhang Huchuan Lu Long Chen VLM 155 9 0 10 Jul 2024
QSync: Quantization-Minimized Synchronous Distributed Training Across Hybrid Devices Juntao Zhao Borui Wan Size Zheng Haibin Lin Yibo Zhu Chuan Wu 132 3 0 02 Jul 2024
Towards Federated Learning with On-device Training and Communication in 8-bit Floating Point Bokun Wang Axel Berg D. A. E. Acar Chuteng Zhou MQ FedML 257 1 0 02 Jul 2024
Optimizing Foundation Model Inference on a Many-tiny-core Open-source RISC-V Platform Viviane Potocnik Luca Colagrande Tim Fischer L. Bertaccini Daniele Jahier Pagliari Luca Bompani Luca Benini 169 3 0 29 May 2024
To FP8 and Back Again: Quantifying Reduced Precision Effects on LLM Training Stability Joonhyung Lee Jeongin Bae Byeongwook Kim S. Kwon Dongsoo Lee MQ 151 4 0 29 May 2024
LoQT: Low Rank Adapters for Quantized Training Sebastian Loeschcke M. Toftrup M. Kastoryano Serge Belongie Vésteinn Snæbjarnarson MQ 157 7 0 26 May 2024
Collage: Light-Weight Low-Precision Strategy for LLM TrainingInternational Conference on Machine Learning (ICML), 2024 Tao Yu Gaurav Gupta Karthick Gopalswamy Amith R. Mamidala Hao Zhou Jeffrey Huynh Youngsuk Park Ron Diamant Hao Ding Jun Huan MQ 161 7 0 06 May 2024
Training-free Graph Neural Networks and the Power of Labels as Features Ryoma Sato 178 5 0 30 Apr 2024
Lightweight Deep Learning for Resource-Constrained Environments: A Survey Hou-I Liu Marco Galindo Hongxia Xie Lai-Kuan Wong Hong-Han Shuai Yung-Hui Li Wen-Huang Cheng 229 126 0 08 Apr 2024
Tiny Machine Learning: Progress and Futures Ji Lin Ligeng Zhu Wei-Ming Chen Wei-Chen Wang Song Han 178 100 0 28 Mar 2024
Hawk: Accurate and Fast Privacy-Preserving Machine Learning Using Secure Lookup Table Computation Hamza Saleem Amir Ziashahabi Muhammad Naveed A. Avestimehr 79 4 0 26 Mar 2024
DaCapo: Accelerating Continuous Learning in Autonomous Systems for Video Analytics Yoonsung Kim Changhun Oh Jinwoo Hwang Wonung Kim Seongryong Oh Yubin Lee Hardik Sharma Amir Yazdanbakhsh Jongse Park 247 13 0 21 Mar 2024
Jetfire: Efficient and Accurate Transformer Pretraining with INT8 Data Flow and Per-Block Quantization Haocheng Xi Yuxiang Chen Kang Zhao Kaijun Zheng Jianfei Chen Jun Zhu MQ 182 27 0 19 Mar 2024
Stochastic Rounding Implicitly Regularizes Tall-and-Thin MatricesSIAM Journal on Matrix Analysis and Applications (SIMAX), 2024 Gregory Dexter Christos Boutsikas Linkai Ma Ilse C. F. Ipsen P. Drineas 115 4 0 18 Mar 2024
IM-Unpack: Training and Inference with Arbitrarily Low Precision IntegersInternational Conference on Machine Learning (ICML), 2024 Zhanpeng Zeng Karthikeyan Sankaralingam Vikas Singh 168 1 0 12 Mar 2024