Root Mean Square Layer Normalization

16 October 2019

Papers citing "Root Mean Square Layer Normalization"

32 / 132 papers shown

Title
ReLU $^2$ Wins: Discovering Efficient Activation Functions for Sparse LLMs Zhengyan Zhang Yixin Song Guanghui Yu Xu Han Yankai Lin Chaojun Xiao Chenyang Song Zhiyuan Liu Zeyu Mi Maosong Sun 22 31 0 06 Feb 2024
TeenyTinyLlama: open-source tiny language models trained in Brazilian Portuguese N. Corrêa Sophia Falk Shiza Fatimah Aniket Sen N. D. Oliveira 30 9 0 30 Jan 2024
PIXAR: Auto-Regressive Language Modeling in Pixel Space Yintao Tai Xiyang Liao Alessandro Suglia Antonio Vergari MLLM 26 7 0 06 Jan 2024
How Smooth Is Attention? Valérie Castin Pierre Ablin Gabriel Peyré AAML 40 9 0 22 Dec 2023
EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism Yanxi Chen Xuchen Pan Yaliang Li Bolin Ding Jingren Zhou LRM 41 30 0 08 Dec 2023
Secure Transformer Inference Protocol Mu Yuan Lan Zhang Xiang-Yang Li 35 3 0 14 Nov 2023
FLTrojan: Privacy Leakage Attacks against Federated Language Models Through Selective Weight Tampering Md. Rafi Ur Rashid Vishnu Asutosh Dasu Kang Gu Najrin Sultana Shagufta Mehnaz AAML FedML 46 10 0 24 Oct 2023
GTA: A Geometry-Aware Attention Mechanism for Multi-View Transformers Takeru Miyato Bernhard Jaeger Max Welling Andreas Geiger ViT 39 15 0 16 Oct 2023
Universal Visual Decomposer: Long-Horizon Manipulation Made Easy Zichen Zhang Yunshuang Li Osbert Bastani Abhishek Gupta Dinesh Jayaraman Yecheng Jason Ma Luca Weihs 37 17 0 12 Oct 2023
Transformer-VQ: Linear-Time Transformers via Vector Quantization Albert Mohwald 34 15 0 28 Sep 2023
Baichuan 2: Open Large-scale Language Models Ai Ming Yang Bin Xiao Bingning Wang Borong Zhang Ce Bian ... Youxin Jiang Yuchen Gao Yupeng Zhang Zenan Zhou Zhiying Wu ELM LRM 77 710 0 19 Sep 2023
ImageBind-LLM: Multi-modality Instruction Tuning Jiaming Han Renrui Zhang Wenqi Shao Peng Gao Peng Xu ... Yafei Wen Xiaoxin Chen Xiangyu Yue Hongsheng Li Yu Qiao MLLM 49 117 0 07 Sep 2023
Norm Tweaking: High-performance Low-bit Quantization of Large Language Models Liang Li Qingyuan Li Bo-Wen Zhang Xiangxiang Chu MQ 47 29 0 06 Sep 2023
Composable Function-preserving Expansions for Transformer Architectures Andrea Gesmundo Kaitlin Maile AI4CE 40 8 0 11 Aug 2023
RecycleGPT: An Autoregressive Language Model with Recyclable Module Yu Jiang Qiaozhi He Xiaomin Zhuang Zhihua Wu Kunpeng Wang Wenlai Zhao Guangwen Yang KELM 28 3 0 07 Aug 2023
On Efficient Training of Large-Scale Deep Learning Models: A Literature Review Li Shen Yan Sun Zhiyuan Yu Liang Ding Xinmei Tian Dacheng Tao VLM 30 41 0 07 Apr 2023
Unlocking the Potential of ChatGPT: A Comprehensive Exploration of its Applications, Advantages, Limitations, and Future Directions in Natural Language Processing Walid Hariri AI4MH LM&MA 33 85 0 27 Mar 2023
Soft Actor-Critic Algorithm with Truly-satisfied Inequality Constraint Taisuke Kobayashi 51 3 0 08 Mar 2023
LLaMA: Open and Efficient Foundation Language Models Hugo Touvron Thibaut Lavril Gautier Izacard Xavier Martinet Marie-Anne Lachaux ... Faisal Azhar Aurelien Rodriguez Armand Joulin Edouard Grave Guillaume Lample ALM PILM 64 12,418 0 27 Feb 2023
Language-Driven Representation Learning for Robotics Siddharth Karamcheti Suraj Nair Annie S. Chen Thomas Kollar Chelsea Finn Dorsa Sadigh Percy Liang LM&Ro SSL 47 145 0 24 Feb 2023
Scaling Vision Transformers to 22 Billion Parameters Mostafa Dehghani Josip Djolonga Basil Mustafa Piotr Padlewski Jonathan Heek ... Mario Luvcić Xiaohua Zhai Daniel Keysers Jeremiah Harmsen N. Houlsby MLLM 90 572 0 10 Feb 2023
Latent Diffusion for Language Generation Justin Lovelace Varsha Kishore Chao-gang Wan Eliot Shekhtman Kilian Q. Weinberger DiffM 29 71 0 19 Dec 2022
Inductive Attention for Video Action Anticipation Tsung-Ming Tai G. Fiameni Cheng-Kuang Lee Simon See Oswald Lanz 39 1 0 17 Dec 2022
The Devil in Linear Transformer Zhen Qin Xiaodong Han Weixuan Sun Dongxu Li Lingpeng Kong Nick Barnes Yiran Zhong 36 70 0 19 Oct 2022
Uncertainty Estimation for Language Reward Models Adam Gleave G. Irving UQLM 39 31 0 14 Mar 2022
ST-MoE: Designing Stable and Transferable Sparse Expert Models Barret Zoph Irwan Bello Sameer Kumar Nan Du Yanping Huang J. Dean Noam M. Shazeer W. Fedus MoE 24 182 0 17 Feb 2022
StyleSwin: Transformer-based GAN for High-resolution Image Generation Bo Zhang Shuyang Gu Bo Zhang Jianmin Bao Dong Chen Fang Wen Yong Wang B. Guo ViT 38 223 0 20 Dec 2021
Improving language models by retrieving from trillions of tokens Sebastian Borgeaud A. Mensch Jordan Hoffmann Trevor Cai Eliza Rutherford ... Simon Osindero Karen Simonyan Jack W. Rae Erich Elsen Laurent Sifre KELM RALM 90 1,027 0 08 Dec 2021
Primer: Searching for Efficient Transformers for Language Modeling David R. So Wojciech Mañke Hanxiao Liu Zihang Dai Noam M. Shazeer Quoc V. Le VLM 91 153 0 17 Sep 2021
Query-Key Normalization for Transformers Alex Henry Prudhvi Raj Dachapally S. Pawar Yuxuan Chen 17 77 0 08 Oct 2020
Group Whitening: Balancing Learning Efficiency and Representational Capacity Lei Huang Yi Zhou Li Liu Fan Zhu Ling Shao 28 21 0 28 Sep 2020
Transformers without Tears: Improving the Normalization of Self-Attention Toan Q. Nguyen Julian Salazar 38 224 0 14 Oct 2019