GLU Variants Improve Transformer

12 February 2020

Papers citing "GLU Variants Improve Transformer"

50 / 647 papers shown

Title
Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretraining Dongyang Liu Shitian Zhao Le Zhuo Weifeng Lin Yu Qiao Xinyue Li Qi Qin Yu Qiao Hongsheng Li Peng Gao MLLM 62 48 0 05 Aug 2024
Penzai + Treescope: A Toolkit for Interpreting, Visualizing, and Editing Models As Data Mingshu Li 33 3 0 01 Aug 2024
Gemma 2: Improving Open Language Models at a Practical Size Gemma Team Gemma Team Morgane Riviere Shreya Pathak Pier Giuseppe Sessa Cassidy Hardin ... Noah Fiedel Armand Joulin Kathleen Kenealy Robert Dadashi Alek Andreev VLM MoE OSLM 37 645 0 31 Jul 2024
UniProcessor: A Text-induced Unified Low-level Image Processor Huiyu Duan Xiongkuo Min Sijing Wu Wei Shen Guangtao Zhai DiffM 39 8 0 30 Jul 2024
A federated large language model for long-term time series forecasting Raed Abdel Sater A. B. Hamza AI4TS 27 2 0 30 Jul 2024
mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval Xin Zhang Yanzhao Zhang Dingkun Long Wen Xie Ziqi Dai ... Pengjun Xie Fei Huang Meishan Zhang Wenjie Li Min Zhang 35 73 0 29 Jul 2024
Enhancing Model Performance: Another Approach to Vision-Language Instruction Tuning Vedanshu M. M. Tripathi Bhavnesh Jaint MLLM VLM 32 0 0 25 Jul 2024
How Lightweight Can A Vision Transformer Be Jen Hong Tan ViT MoE 57 0 0 25 Jul 2024
$u-$\mu$P: The Unit-Scaled Maximal Update Parametrization$ u- $\mu$ P: The Unit-Scaled Maximal Update Parametrization Charlie Blake C. Eichenberg Josef Dean Lukas Balles Luke Y. Prince Bjorn Deiseroth Andres Felipe Cruz Salinas Carlo Luschi Samuel Weinbach Douglas Orr 51 9 0 24 Jul 2024
Stretching Each Dollar: Diffusion Training from Scratch on a Micro-Budget Vikash Sehwag Xianghao Kong Jingtao Li Michael Spranger Lingjuan Lyu DiffM 39 9 0 22 Jul 2024
Inverted Activations Georgii Sergeevich Novikov Ivan V. Oseledets 21 0 0 22 Jul 2024
Beyond Next Token Prediction: Patch-Level Training for Large Language Models Chenze Shao Fandong Meng Jie Zhou 41 1 0 17 Jul 2024
Any-Property-Conditional Molecule Generation with Self-Criticism using Spanning Trees Alexia Jolicoeur-Martineau A. Baratin Kisoo Kwon Boris Knyazev Yan Zhang 36 1 0 12 Jul 2024
Flash normalization: fast normalization for LLMs Nils Graef Matthew Clapp Andrew Wasielewski 19 0 0 12 Jul 2024
RoLoRA: Fine-tuning Rotated Outlier-free LLMs for Effective Weight-Activation Quantization Xijie Huang Zechun Liu Shih-yang Liu Kwang-Ting Cheng MQ 35 7 0 10 Jul 2024
Toto: Time Series Optimized Transformer for Observability Ben Cohen E. Khwaja Kan Wang Charles Masson Elise Ramé Youssef Doubli Othmane Abou-Amal AI4TS 38 3 0 10 Jul 2024
How Effective are State Space Models for Machine Translation? Hugo Pitorro Pavlo Vasylenko Marcos Vinícius Treviso André F. T. Martins Mamba 43 2 0 07 Jul 2024
YourMT3+: Multi-instrument Music Transcription with Enhanced Transformer Architectures and Cross-dataset Stem Augmentation Sungkyun Chang Emmanouil Benetos Holger Kirchhoff Simon Dixon 29 2 0 05 Jul 2024
Learning to (Learn at Test Time): RNNs with Expressive Hidden States Yu Sun Xinhao Li Karan Dalal Jiarui Xu Arjun Vikram ... Xinlei Chen Xiaolong Wang Sanmi Koyejo Tatsunori Hashimoto Carlos Guestrin 56 92 0 05 Jul 2024
Mixture of A Million Experts Xu Owen He MoE 31 25 0 04 Jul 2024
The Mysterious Case of Neuron 1512: Injectable Realignment Architectures Reveal Internal Characteristics of Meta's Llama 2 Model Brenden Smith Dallin Baker Clayton Chase Myles Barney Kaden Parker Makenna Allred Peter Hu Alex Evans Nancy Fulda 32 0 0 04 Jul 2024
Enhancing Translation Accuracy of Large Language Models through Continual Pre-Training on Parallel Data Minato Kondo T. Utsuro Masaaki Nagata CLL 36 4 0 03 Jul 2024
52B to 1T: Lessons Learned via Tele-FLM Series Xiang Li Yiqun Yao Xin Jiang Xuezhi Fang Chao Wang ... Yequan Wang Zhongjiang He Zhongyuan Wang Xuelong Li Tiejun Huang ALM LRM 39 2 0 03 Jul 2024
Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs Enshu Liu Junyi Zhu Zinan Lin Xuefei Ning Matthew B. Blaschko Shengen Yan Guohao Dai Huazhong Yang Yu Wang MoE 54 5 0 01 Jul 2024
YuLan: An Open-source Large Language Model Yutao Zhu Kun Zhou Kelong Mao Wentong Chen Yiding Sun ... Wenbing Huang Ze-Feng Gao Yueguo Chen Weizheng Lu Ji-Rong Wen ALM ELM 42 0 0 28 Jun 2024
Resolving Discrepancies in Compute-Optimal Scaling of Language Models Tomer Porian Mitchell Wortsman J. Jitsev Ludwig Schmidt Y. Carmon 50 20 0 27 Jun 2024
Grass: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradients Aashiq Muhamed Oscar Li David Woodruff Mona Diab Virginia Smith 45 7 0 25 Jun 2024
LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training Tong Zhu Xiaoye Qu Daize Dong Jiacheng Ruan Jingqi Tong Conghui He Yu Cheng MoE ALM 46 71 0 24 Jun 2024
OTCE: Hybrid SSM and Attention with Cross Domain Mixture of Experts to construct Observer-Thinker-Conceiver-Expresser Jingze Shi Ting Xie Bingheng Wu Chunjun Zheng Kai Wang 22 2 0 24 Jun 2024
Reducing Fine-Tuning Memory Overhead by Approximate and Memory-Sharing Backpropagation Yuchen Yang Yingdong Shi Cheems Wang Xiantong Zhen Yuxuan Shi Jun Xu 32 1 0 24 Jun 2024
Unsupervised Extraction of Dialogue Policies from Conversations Makesh Narsimhan Sreedhar Traian Rebedea Christopher Parisien OffRL 18 2 0 21 Jun 2024
RouteFinder: Towards Foundation Models for Vehicle Routing Problems Federico Berto Chuanbo Hua Nayeli Gast Zepeda André Hottung N. Wouda Leon Lan Kevin Tierney J. Park Jinkyoo Park 48 10 0 21 Jun 2024
ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools Team GLM : Aohan Zeng Bin Xu Bowen Wang ... Zhaoyu Wang Zhen Yang Zhengxiao Du Zhenyu Hou Zihan Wang ALM 65 477 0 18 Jun 2024
MCSD: An Efficient Language Model with Diverse Fusion Hua Yang Duohai Li Shiman Li 27 2 0 18 Jun 2024
LiLiuM: eBay's Large Language Models for e-commerce Christian Herold Michael Kozielski Leonid Ekimov Pavel Petrushkov P. Vandenbussche Shahram Khadivi 35 1 0 17 Jun 2024
Optimized Speculative Sampling for GPU Hardware Accelerators Dominik Wagner Seanie Lee Ilja Baumann Philipp Seeberger K. Riedhammer Tobias Bocklet 38 3 0 16 Jun 2024
H-Fac: Memory-Efficient Optimization with Factorized Hamiltonian Descent Son Nguyen Lizhang Chen Bo Liu Qiang Liu 20 3 0 14 Jun 2024
GEB-1.3B: Open Lightweight Large Language Model Jie Wu Yufeng Zhu Lei Shen Xuqing Lu ALM 29 0 0 14 Jun 2024
Alleviating Distortion in Image Generation via Multi-Resolution Diffusion Models Qihao Liu Zhanpeng Zeng Ju He Qihang Yu Xiaohui Shen Liang-Chieh Chen 46 18 0 13 Jun 2024
4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities Roman Bachmann Oğuzhan Fatih Kar David Mizrahi Ali Garjani Mingfei Gao David Griffiths Jiaming Hu Afshin Dehghan Amir Zamir MoE VLM MLLM 36 14 0 13 Jun 2024
An Empirical Study of Mamba-based Language Models R. Waleffe Wonmin Byeon Duncan Riach Brandon Norick V. Korthikanti ... Vartika Singh Jared Casper Jan Kautz M. Shoeybi Bryan Catanzaro 54 64 0 12 Jun 2024
Fetch-A-Set: A Large-Scale OCR-Free Benchmark for Historical Document Retrieval Adrià Molina O. R. Terrades Josep Lladós 28 0 0 11 Jun 2024
Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling Liliang Ren Yang Liu Yadong Lu Yelong Shen Chen Liang Weizhu Chen Mamba 64 55 0 11 Jun 2024
Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation Peize Sun Yi Jiang Shoufa Chen Shilong Zhang Bingyue Peng Ping Luo Zehuan Yuan VLM 60 221 0 10 Jun 2024
PowerInfer-2: Fast Large Language Model Inference on a Smartphone Zhenliang Xue Yixin Song Zeyu Mi Le Chen Yubin Xia Haibo Chen 46 42 0 10 Jun 2024
Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameters Yixin Song Haotong Xie Zhengyan Zhang Bo Wen Li Ma Zeyu Mi Haibo Chen MoE 29 21 0 10 Jun 2024
Attention as a Hypernetwork Simon Schug Seijin Kobayashi Yassir Akram João Sacramento Razvan Pascanu GNN 33 3 0 09 Jun 2024
Accelerating evolutionary exploration through language model-based transfer learning M. Reissmann Yuan Fang Andrew S. H. Ooi R. D. Sandberg 34 2 0 07 Jun 2024
Phy-Diff: Physics-guided Hourglass Diffusion Model for Diffusion MRI Synthesis Juanhua Zhang Ruodan Yan Alessandro Perelli Xi Chen Chao Li MedIm DiffM 48 5 0 05 Jun 2024
Xmodel-LM Technical Report Yichuan Wang Yang Liu Yu Yan Qun Wang Xucheng Huang Ling Jiang OSLM ALM 27 1 0 05 Jun 2024