Root Mean Square Layer Normalization

16 October 2019

Papers citing "Root Mean Square Layer Normalization"

50 / 131 papers shown

Title
Flash-VL 2B: Optimizing Vision-Language Model Performance for Ultra-Low Latency and High Throughput Bo Zhang Shuo Li Runhe Tian Yang Yang Jixin Tang Jinhao Zhou Lin Ma VLM 31 0 0 14 May 2025
Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free Zihan Qiu Zhaoxiang Wang Bo Zheng Zeyu Huang Kaiyue Wen ... Fei Huang Suozhi Huang Dayiheng Liu Jingren Zhou Junyang Lin MoE 40 0 0 10 May 2025
Parameter-Efficient Transformer Embeddings Henry Ndubuaku Mouad Talhi 29 0 0 04 May 2025
Model Connectomes: A Generational Approach to Data-Efficient Language Models Klemen Kotar Greta Tuckute 57 0 0 29 Apr 2025
MicarVLMoE: A Modern Gated Cross-Aligned Vision-Language Mixture of Experts Model for Medical Image Captioning and Report Generation Amaan Izhar Nurul Japar Norisma Idris Ting Dang MoE 73 0 0 29 Apr 2025
ISDrama: Immersive Spatial Drama Generation through Multimodal Prompting Wenjie Qu Wenxiang Guo Changhao Pan Zehan Zhu Tao Jin Zhou Zhao VGen 54 1 0 29 Apr 2025
Towards Robust Multimodal Physiological Foundation Models: Handling Arbitrary Missing Modalities Xi Fu Wei-Bang Jiang Yi Ding Cuntai Guan 46 0 0 28 Apr 2025
A Comparative Study on Positional Encoding for Time-frequency Domain Dual-path Transformer-based Source Separation Models Kohei Saijo Tetsuji Ogawa 52 1 0 28 Apr 2025
CasaGPT: Cuboid Arrangement and Scene Assembly for Interior Design Weitao Feng Hang Zhou Jing Liao Li Cheng Wenbo Zhou 3DV 58 0 0 28 Apr 2025
Versatile Framework for Song Generation with Prompt-based Control Wenjie Qu Wenxiang Guo Changhao Pan Zehan Zhu Ruiqi Li ... Rongjie Huang Ruiyuan Zhang Zhiqing Hong Ziyue Jiang Zhou Zhao 77 2 0 27 Apr 2025
Trillion 7B Technical Report Sungjun Han Juyoung Suk Suyeong An Hyungguk Kim Kyuseok Kim Wonsuk Yang Seungtaek Choi Jamin Shin 167 1 0 21 Apr 2025
Adaptive Layer-skipping in Pre-trained LLMs Xuan Luo Weizhi Wang Xifeng Yan 200 0 0 31 Mar 2025
TRA: Better Length Generalisation with Threshold Relative Attention Mattia Opper Roland Fernandez P. Smolensky Jianfeng Gao 46 0 0 29 Mar 2025
Tiled Flash Linear Attention: More Efficient Linear RNN and xLSTM Kernels M. Beck Korbinian Poppel Phillip Lippe Sepp Hochreiter 69 1 0 18 Mar 2025
Cost-Optimal Grouped-Query Attention for Long-Context Modeling Yuxiao Chen Yutong Wu Chenyang Song Zhiyuan Liu Maosong Sun Xu Han Zhiyuan Liu Maosong Sun 73 0 0 12 Mar 2025
SANA-Sprint: One-Step Diffusion with Continuous-Time Consistency Distillation Junsong Chen Shuchen Xue Yuyang Zhao Jincheng Yu Sayak Paul Junyu Chen Han Cai E. Xie Enze Xie VLM 66 2 0 12 Mar 2025
Patch-Depth Fusion: Dichotomous Image Segmentation via Fine-Grained Patch Strategy and Depth Integrity-Prior Xianjie Liu Keren Fu Qijun Zhao MDE 59 0 0 08 Mar 2025
EuroBERT: Scaling Multilingual Encoders for European Languages Nicolas Boizard Hippolyte Gisserot-Boukhlef Duarte M. Alves André F. T. Martins Ayoub Hammal ... Maxime Peyrard Nuno M. Guerreiro Patrick Fernandes Ricardo Rei Pierre Colombo 175 2 0 07 Mar 2025
How can representation dimension dominate structurally pruned LLMs? Mingxue Xu Lisa Alazraki Danilo Mandic 56 0 0 06 Mar 2025
HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization Zhijian Zhuo Yutao Zeng Ya Wang Sijun Zhang Jian Yang Xiaoqing Li Xun Zhou Jinwen Ma 51 0 0 06 Mar 2025
Is Pre-training Applicable to the Decoder for Dense Prediction? Chao Ning Wanshui Gan Weihao Xuan Naoto Yokoya 48 0 0 05 Mar 2025
FANformer: Improving Large Language Models Through Effective Periodicity Modeling Yihong Dong Ge Li Xue Jiang Yongding Tao Kechi Zhang ... Huanyu Liu Jiazheng Ding Jia Li Jinliang Deng Hong Mei AI4TS 46 0 0 28 Feb 2025
Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization Taishi Nakamura Takuya Akiba Kazuki Fujii Yusuke Oda Rio Yokota Jun Suzuki MoMe MoE 94 1 0 26 Feb 2025
Stable-SPAM: How to Train in 4-Bit More Stably than 16-Bit Adam Tianjin Huang Haotian Hu Zhenyu Zhang Gaojie Jin Xianrui Li ... Tianlong Chen Lu Liu Qingsong Wen Zhangyang Wang Shiwei Liu MQ 41 0 0 24 Feb 2025
Unveiling Reasoning Thresholds in Language Models: Scaling, Fine-Tuning, and Interpretability through Attention Maps Yen-Che Hsiao Abhishek Dutta LRM ReLM ELM 66 0 0 24 Feb 2025
Uncertainty Representations in State-Space Layers for Deep Reinforcement Learning under Partial Observability Carlos E. Luis A. Bottero Julia Vinogradska Felix Berkenkamp Jan Peters 78 1 0 20 Feb 2025
C2T: A Classifier-Based Tree Construction Method in Speculative Decoding Feiye Huo Jianchao Tan Kaipeng Zhang Xunliang Cai Shengli Sun 44 0 0 20 Feb 2025
Baichuan-M1: Pushing the Medical Capability of Large Language Models Binghui Wang Haizhou Zhao Huozhi Zhou Liang Song Mingyu Xu ... Yan Zhang Yifei Duan Yuyan Zhou Zhi-Ming Ma Zhikai Wu LM&MA ELM AI4MH 42 4 0 18 Feb 2025
Large Language Diffusion Models Shen Nie Fengqi Zhu Zebin You Xiaolu Zhang Jingyang Ou Jun Hu Jun Zhou Yankai Lin Zhicheng Dou Chongxuan Li 112 19 0 14 Feb 2025
I Think, Therefore I Diffuse: Enabling Multimodal In-Context Reasoning in Diffusion Models Zhenxing Mi Kuan-Chieh Jackson Wang Guocheng Qian Hanrong Ye Runtao Liu Sergey Tulyakov Kfir Aberman Dan Xu LRM 47 0 0 12 Feb 2025
Life-Code: Central Dogma Modeling with Multi-Omics Sequence Unification Zicheng Liu Siyuan Li Zhiyuan Chen Lei Xin Fang Wu Chang Yu Qirong Yang Yucheng Guo Yifan Yang Stan Z. Li SyDa AI4CE 92 0 0 11 Feb 2025
Steel-LLM:From Scratch to Open Source -- A Personal Journey in Building a Chinese-Centric LLM Qingshui Gu Shu Li Tianyu Zheng Zhaoxiang Zhang 258 0 0 10 Feb 2025
$FuXi-$\alpha$: Scaling Recommendation Model with Feature Interaction Enhanced Transformer$ FuXi- $\alpha$ : Scaling Recommendation Model with Feature Interaction Enhanced Transformer Yufei Ye Wei Guo Jin Yao Chin Hao Wang Hong Zhu ... Yuyang Ye Y. Liu Ruiming Tang Defu Lian Enhong Chen 104 2 0 05 Feb 2025
Rethinking Homogeneity of Vision and Text Tokens in Large Vision-and-Language Models Chia-Wen Kuo Sijie Zhu Fan Chen Xiaohui Shen Longyin Wen VLM 65 1 0 04 Feb 2025
Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment Gregor Bachmann Sotiris Anagnostidis Albert Pumarola Markos Georgopoulos A. Sanakoyeu Yuming Du Edgar Schönfeld Ali K. Thabet Jonas Kohler ALM BDL 106 8 0 31 Jan 2025
SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Training Tianjin Huang Ziquan Zhu Gaojie Jin Lu Liu Zhangyang Wang Shiwei Liu 47 1 0 12 Jan 2025
Switti: Designing Scale-Wise Transformers for Text-to-Image Synthesis Anton Voronov Denis Kuznedelev Mikhail Khoroshikh Valentin Khrulkov Dmitry Baranchuk 114 2 0 02 Dec 2024
More Expressive Attention with Negative Weights Ang Lv Ruobing Xie Shuaipeng Li Jiayi Liao Xingchen Sun Zhanhui Kang Di Wang Rui Yan 42 0 0 11 Nov 2024
On Improved Conditioning Mechanisms and Pre-training Strategies for Diffusion Models Tariq Berrada Ifriqi Pietro Astolfi Melissa Hall Reyhane Askari Hemmat Yohann Benchetrit ... Matthew Muckley Karteek Alahari Adriana Romero Soriano Jakob Verbeek M. Drozdzal AI4CE VLM 75 2 0 05 Nov 2024
TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters Haiyang Wang Yue Fan Muhammad Ferjad Naeem Yongqin Xian J. E. Lenssen Liwei Wang F. Tombari Bernt Schiele 49 2 0 30 Oct 2024
MrT5: Dynamic Token Merging for Efficient Byte-level Language Models Julie Kallini Shikhar Murty Christopher D. Manning Christopher Potts Róbert Csordás 40 2 0 28 Oct 2024
Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA Sangmin Bae Adam Fisch Hrayr Harutyunyan Ziwei Ji Seungyeon Kim Tal Schuster KELM 84 5 0 28 Oct 2024
COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Training Haocheng Xi Han Cai Ligeng Zhu Yaojie Lu Kurt Keutzer Jianfei Chen Song Han MQ 75 9 0 25 Oct 2024
Scaling up Masked Diffusion Models on Text Shen Nie Fengqi Zhu Chao Du Tianyu Pang Qian Liu Guangtao Zeng Min Lin Chongxuan Li AI4CE 63 14 0 24 Oct 2024
LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics Thomas Robert M. Safaryan Ionut-Vlad Modoranu Dan Alistarh ODL 36 2 0 21 Oct 2024
SF-Speech: Straightened Flow for Zero-Shot Voice Clone Xuyuan Li Zengqiang Shang Hua Hua Peiyang Shi Chen Yang Li Wang Pengyuan Zhang 55 2 0 16 Oct 2024
Transfer Learning with Foundational Models for Time Series Forecasting using Low-Rank Adaptations M. Germán-Morales A. J. Rivera-Rivas M. J. del Jesus Díaz C. J. Carmona AI4TS AI4CE 56 0 0 15 Oct 2024
Liger Kernel: Efficient Triton Kernels for LLM Training Pin-Lun Hsu Yun Dai Vignesh Kothapalli Qingquan Song Shao Tang Siyu Zhu Steven Shimizu Shivam Sahni Haowen Ning Yanning Chen 53 30 0 14 Oct 2024
ControlMM: Controllable Masked Motion Generation Ekkasit Pinyoanuntapong Muhammad Usama Saleem Korrawe Karunratanakul Pu Wang Hongfei Xue Chong Chen Chuan Guo Junli Cao J. Ren Sergey Tulyakov VGen 37 4 0 14 Oct 2024
DARE the Extreme: Revisiting Delta-Parameter Pruning For Fine-Tuned Models Wenlong Deng Yize Zhao V. Vakilian Minghui Chen Xiaoxiao Li Christos Thrampoulidis 45 3 0 12 Oct 2024