Root Mean Square Layer Normalization

16 October 2019

Papers citing "Root Mean Square Layer Normalization"

50 / 136 papers shown

Title
Transfer Learning with Foundational Models for Time Series Forecasting using Low-Rank Adaptations M. Germán-Morales A. J. Rivera-Rivas M. J. del Jesus Díaz C. J. Carmona AI4TS AI4CE 56 0 0 15 Oct 2024
Liger Kernel: Efficient Triton Kernels for LLM Training Pin-Lun Hsu Yun Dai Vignesh Kothapalli Qingquan Song Shao Tang Siyu Zhu Steven Shimizu Shivam Sahni Haowen Ning Yanning Chen 56 30 0 14 Oct 2024
ControlMM: Controllable Masked Motion Generation Ekkasit Pinyoanuntapong Muhammad Usama Saleem Korrawe Karunratanakul Pu Wang Hongfei Xue Chong Chen Chuan Guo Junli Cao J. Ren Sergey Tulyakov VGen 42 4 0 14 Oct 2024
DARE the Extreme: Revisiting Delta-Parameter Pruning For Fine-Tuned Models Wenlong Deng Yize Zhao V. Vakilian Minghui Chen Xiaoxiao Li Christos Thrampoulidis 50 4 0 12 Oct 2024
Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training Gen Luo Xue Yang Wenhan Dou Zhaokai Wang Jifeng Dai Jifeng Dai Yu Qiao Xizhou Zhu VLM MLLM 73 26 0 10 Oct 2024
RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation Songming Liu Lingxuan Wu Bangguo Li Hengkai Tan Huayu Chen Zhengyi Wang Ke Xu Hang Su Jun Zhu 45 85 0 10 Oct 2024
Stuffed Mamba: State Collapse and State Capacity of RNN-Based Long-Context Modeling Yingfa Chen Xinrong Zhang Shengding Hu Xu Han Zhiyuan Liu Maosong Sun Mamba 59 2 0 09 Oct 2024
Rodimus*: Breaking the Accuracy-Efficiency Trade-Off with Efficient Attentions Zhihao He Hang Yu Zi Gong Shizhan Liu Jia-Nan Li Weiyao Lin VLM 38 1 0 09 Oct 2024
From Tokens to Words: On the Inner Lexicon of LLMs Guy Kaplan Matanel Oren Yuval Reif Roy Schwartz 61 14 0 08 Oct 2024
Differential Transformer Tianzhu Ye Li Dong Yuqing Xia Yutao Sun Yi Zhu Gao Huang Furu Wei 236 0 0 07 Oct 2024
Large Language Models as Markov Chains Oussama Zekri Ambroise Odonnat Abdelhakim Benechehab Linus Bleistein Nicolas Boullé I. Redko 55 10 0 03 Oct 2024
Selective Attention Improves Transformer Yaniv Leviathan Matan Kalman Yossi Matias 51 9 0 03 Oct 2024
On Expressive Power of Looped Transformers: Theoretical Analysis and Enhancement via Timestep Encoding Kevin Xu Issei Sato 41 3 0 02 Oct 2024
Characterizing and Efficiently Accelerating Multimodal Generation Model Inference Yejin Lee Anna Y. Sun Basil Hosmer Bilge Acun Can Balioglu ... Ram Pasunuru Scott Yih Sravya Popuri Xing Liu Carole-Jean Wu 57 2 0 30 Sep 2024
Hyper-Connections Defa Zhu Hongzhi Huang Zihao Huang Yutao Zeng Yunyao Mao Banggu Wu Qiyang Min Xun Zhou 41 4 0 29 Sep 2024
LiRA: Light-Robust Adversary for Model-based Reinforcement Learning in Real World Taisuke Kobayashi 71 2 0 29 Sep 2024
Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts Xiaoming Shi Shiyu Wang Yuqi Nie Dianqi Li Zhou Ye Qingsong Wen Ming Jin AI4TS 53 32 0 24 Sep 2024
EMMeTT: Efficient Multimodal Machine Translation Training Piotr Żelasko Zhehuai Chen Mengru Wang Daniel Galvez Oleksii Hrinchuk Shuoyang Ding Ke Hu Jagadeesh Balam Vitaly Lavrukhin Boris Ginsburg 42 1 0 20 Sep 2024
Chain-of-Thought Prompting for Speech Translation Ke Hu Zhehuai Chen Chao-Han Huck Yang Piotr Żelasko Oleksii Hrinchuk Vitaly Lavrukhin Jagadeesh Balam Boris Ginsburg LRM 41 3 0 17 Sep 2024
Flash STU: Fast Spectral Transform Units Y. Isabel Liu Windsor Nguyen Yagiz Devre Evan Dogariu Anirudha Majumdar Elad Hazan AI4TS 72 1 0 16 Sep 2024
MambaFoley: Foley Sound Generation using Selective State-Space Models Marco Furio Colombo Francesca Ronchini Luca Comanducci Fabio Antonacci Mamba 25 1 0 13 Sep 2024
SELD-Mamba: Selective State-Space Model for Sound Event Localization and Detection with Source Distance Estimation Da Mu Zhicheng Zhang Haobo Yue Zehao Wang Jin Tang Jianqin Yin Mamba 51 1 0 09 Aug 2024
Diffusion Guided Language Modeling Justin Lovelace Varsha Kishore Yiwei Chen Kilian Q. Weinberger 44 6 0 08 Aug 2024
Table Transformers for Imputing Textual Attributes Ting-Ruen Wei Yuan Wang Yoshitaka Inoue Hsin-Tai Wu Yi Fang LMTD 45 0 0 04 Aug 2024
Penzai + Treescope: A Toolkit for Interpreting, Visualizing, and Editing Models As Data Mingshu Li 46 3 0 01 Aug 2024
$u-$\mu$P: The Unit-Scaled Maximal Update Parametrization$ u- $\mu$ P: The Unit-Scaled Maximal Update Parametrization Charlie Blake C. Eichenberg Josef Dean Lukas Balles Luke Y. Prince Bjorn Deiseroth Andres Felipe Cruz Salinas Carlo Luschi Samuel Weinbach Douglas Orr 61 10 0 24 Jul 2024
RazorAttention: Efficient KV Cache Compression Through Retrieval Heads Hanlin Tang Yang Lin Jing Lin Qingsen Han Shikuan Hong Yiwu Yao Gongyi Wang MQ 44 27 0 22 Jul 2024
Beyond Next Token Prediction: Patch-Level Training for Large Language Models Chenze Shao Fandong Meng Jie Zhou 53 1 0 17 Jul 2024
Any-Property-Conditional Molecule Generation with Self-Criticism using Spanning Trees Alexia Jolicoeur-Martineau A. Baratin Kisoo Kwon Boris Knyazev Yan Zhang 45 1 0 12 Jul 2024
Controlling Space and Time with Diffusion Models Daniel Watson Saurabh Saxena Lala Li Andrea Tagliasacchi David J. Fleet VGen 71 27 0 10 Jul 2024
Video In-context Learning: Autoregressive Transformers are Zero-Shot Video Imitators Wentao Zhang Junliang Guo Tianyu He Li Zhao Linli Xu Jiang Bian 47 3 0 10 Jul 2024
Learning to (Learn at Test Time): RNNs with Expressive Hidden States Yu Sun Xinhao Li Karan Dalal Jiarui Xu Arjun Vikram ... Xinlei Chen Xiaolong Wang Sanmi Koyejo Tatsunori Hashimoto Carlos Guestrin 73 93 0 05 Jul 2024
Reducing Fine-Tuning Memory Overhead by Approximate and Memory-Sharing Backpropagation Yuchen Yang Yingdong Shi Cheems Wang Xiantong Zhen Yuxuan Shi Jun Xu 40 1 0 24 Jun 2024
RouteFinder: Towards Foundation Models for Vehicle Routing Problems Federico Berto Chuanbo Hua Nayeli Gast Zepeda André Hottung N. Wouda Leon Lan Kevin Tierney J. Park Jinkyoo Park 61 10 0 21 Jun 2024
Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling Liliang Ren Yang Liu Yadong Lu Yelong Shen Chen Liang Weizhu Chen Mamba 77 57 0 11 Jun 2024
Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation Peize Sun Yi Jiang Shoufa Chen Shilong Zhang Bingyue Peng Ping Luo Zehuan Yuan VLM 68 230 0 10 Jun 2024
Attention as a Hypernetwork Simon Schug Seijin Kobayashi Yassir Akram João Sacramento Razvan Pascanu GNN 37 3 0 09 Jun 2024
Accelerating evolutionary exploration through language model-based transfer learning M. Reissmann Yuan Fang Andrew S. H. Ooi R. D. Sandberg 47 2 0 07 Jun 2024
Understanding and Minimising Outlier Features in Neural Network Training Bobby He Lorenzo Noci Daniele Paliotta Imanol Schlag Thomas Hofmann 47 3 0 29 May 2024
BiSup: Bidirectional Quantization Error Suppression for Large Language Models Minghui Zou Ronghui Guo Sai Zhang Xiaowang Zhang Zhiyong Feng MQ 44 1 0 24 May 2024
iVideoGPT: Interactive VideoGPTs are Scalable World Models Jialong Wu Shaofeng Yin Ningya Feng Xu He Dong Li Haifeng Zhang Mingsheng Long VGen 54 26 0 24 May 2024
TerDiT: Ternary Diffusion Models with Transformers Xudong Lu Aojun Zhou Ziyi Lin Qi Liu Yuhui Xu Renrui Zhang Yafei Wen Shuai Ren Peng Gao Junchi Yan MQ 55 2 0 23 May 2024
Chameleon: Mixed-Modal Early-Fusion Foundation Models Chameleon Team MLLM 62 266 0 16 May 2024
Granite Code Models: A Family of Open Foundation Models for Code Intelligence Mayank Mishra Matt Stallone Gaoyuan Zhang Songlin Yang Aditya Prasad ... Amith Singhee Nirmit Desai David D. Cox Ruchir Puri Yikang Shen AI4TS 65 58 0 07 May 2024
EEGEncoder: Advancing BCI with Transformer-Based Motor Imagery Classification Wangdan Liao Weidong Wang 28 4 0 23 Apr 2024
Stable Code Technical Report Nikhil Pinnaparaju Reshinth Adithyan Duy Phung J. Tow James Baicoianu ... Maksym Zhuravinskyi Dakota Mahan Marco Bellagente Carlos Riquelme Nathan Cooper LRM ALM 25 13 0 01 Apr 2024
Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference Han Zhao Min Zhang Wei Zhao Pengxiang Ding Siteng Huang Donglin Wang Mamba 54 69 0 21 Mar 2024
Dissecting Deep RL with High Update Ratios: Combatting Value Divergence Marcel Hussing C. Voelcker Igor Gilitschenski Amir-massoud Farahmand Eric Eaton 47 3 0 09 Mar 2024
DeepSeek-VL: Towards Real-World Vision-Language Understanding Haoyu Lu Wen Liu Bo Zhang Bing-Li Wang Kai Dong ... Yaofeng Sun Chengqi Deng Hanwei Xu Zhenda Xie Chong Ruan VLM 41 304 0 08 Mar 2024
GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection Jiawei Zhao Zhenyu Zhang Beidi Chen Zhangyang Wang A. Anandkumar Yuandong Tian 45 179 0 06 Mar 2024