139

Flash normalization: fast RMSNorm for LLMs

Main:8 Pages
14 Figures
Bibliography:2 Pages
Abstract

RMSNorm is used by many LLMs such as Llama, Mistral, and OpenELM. This paper details FlashNorm, which is an exact but faster implementation of RMSNorm followed by linear layers. See this https URL for code and more transformer tricks.

View on arXiv
Comments on this paper