Decoder-only transformer networks have become incredibly popular for language modeling tasks. State-of-the-art models can have over a hundred transformer blocks, containing billions of trainable parameters, and are trained on trillions of tokens of text. Each transformer block typically consists of a multi-head attention (MHA) mechanism and a two-layer fully connected feedforward network (FFN). In this paper, we examine the importance of the FFN during the model pre-training process through a series of experiments, confirming that the FFN is important to model performance. Furthermore, we show that models using a transformer block configuration with three-layer FFNs with fewer such blocks outperform the standard two-layer configuration delivering lower training loss with fewer total parameters in less time.
View on arXiv@article{gerber2025_2505.06633,
title={ Attention Is Not All You Need: The Importance of Feedforward Networks in Transformer Models },
author={ Isaac Gerber },
journal={arXiv preprint arXiv:2505.06633},
year={ 2025 }
}