Mixture-of-Depths: Dynamically allocating compute in transformer-based language models

2 April 2024

Papers citing "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"

50 / 51 papers shown

Title
ORXE: Orchestrating Experts for Dynamically Configurable Efficiency Qingyuan Wang Guoxin Wang B. Cardiff Deepu John 31 0 0 07 May 2025
Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routing Piotr Piekos Róbert Csordás Jürgen Schmidhuber MoE VLM 94 1 0 01 May 2025
Revisiting Transformers through the Lens of Low Entropy and Dynamic Sparsity Ruifeng Ren Yong Liu 63 0 0 26 Apr 2025
SpecEE: Accelerating Large Language Model Inference with Speculative Early Exiting Jiaming Xu Jiayi Pan Yongkang Zhou Siming Chen J. Li Yaoxiu Lian Junyi Wu Guohao Dai LRM 33 0 0 11 Apr 2025
Adaptive Layer-skipping in Pre-trained LLMs Xuan Luo Weizhi Wang Xifeng Yan 95 0 0 31 Mar 2025
MoLe-VLA: Dynamic Layer-skipping Vision Language Action Model via Mixture-of-Layers for Efficient Robot Manipulation Rongyu Zhang Menghang Dong Yuan Zhang Liang Heng Xiaowei Chi Gaole Dai Li Du Dan Wang Yuan Du MoE 81 0 0 26 Mar 2025
Learning to Inference Adaptively for Multimodal Large Language Models Zhuoyan Xu Khoi Duc Nguyen Preeti Mukherjee Saurabh Bagchi Somali Chaterji Yingyu Liang Yin Li LRM 42 1 0 13 Mar 2025
Discovering Influential Neuron Path in Vision Transformers Yifan Wang Yifei Liu Yingdong Shi C. Li Anqi Pang Sibei Yang Jingyi Yu Kan Ren ViT 69 0 0 12 Mar 2025
Exposure Bias Reduction for Enhancing Diffusion Transformer Feature Caching Zhen Zou Hu Yu Jie Xiao Feng Zhao 40 0 0 10 Mar 2025
LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistant Wei Li Bing Hu Rui Shao Leyang Shen Liqiang Nie 39 2 0 05 Mar 2025
Unifying Light Field Perception with Field of Parallax Fei Teng Buyin Deng Boyuan Zheng Kai Luo Kunyu Peng Jiaming Zhang Kailun Yang 34 0 0 02 Mar 2025
A Sliding Layer Merging Method for Efficient Depth-Wise Pruning in LLMs Xuan Ding Rui Sun Yunjian Zhang Xiu Yan Yueqi Zhou Kaihao Huang Suzhong Fu Angelica I Aviles-Rivero Chuanlong Xie Yao Zhu 84 1 0 26 Feb 2025
Ray-Tracing for Conditionally Activated Neural Networks Claudio Gallicchio Giuseppe Nuti AI4CE 55 0 0 21 Feb 2025
UniMoD: Efficient Unified Multimodal Transformers with Mixture-of-Depths Weijia Mao Z. Yang Mike Zheng Shou MoE 65 0 0 10 Feb 2025
M2R2: Mixture of Multi-Rate Residuals for Efficient Transformer Inference Nikhil Bhendawade Mahyar Najibi Devang Naik Irina Belousova MoE 85 0 0 04 Feb 2025
Adaptive Self-improvement LLM Agentic System for ML Library Development Genghan Zhang Weixin Liang Olivia Hsu K. Olukotun 95 0 0 04 Feb 2025
Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models Z. Qiu Zeyu Huang Bo Zheng Kaiyue Wen Z. Wang Rui Men Ivan Titov Dayiheng Liu Jingren Zhou Junyang Lin MoE 49 5 0 21 Jan 2025
Complexity Experts are Task-Discriminative Learners for Any Image Restoration Eduard Zamfir Zongwei Wu Nancy Mehta Yuedong Tan Danda Pani Paudel Yulun Zhang Radu Timofte MoE 121 1 0 27 Nov 2024
ShowUI: One Vision-Language-Action Model for GUI Visual Agent Kevin Qinghong Lin Linjie Li Difei Gao Z. Yang Shiwei Wu Zechen Bai Weixian Lei Lijuan Wang Mike Zheng Shou LLMAG 72 13 0 26 Nov 2024
DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Execution Yang Yue Yulin Wang Bingyi Kang Yizeng Han Shenzhi Wang Shiji Song Jiashi Feng Gao Huang VLM 40 16 0 04 Nov 2024
Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA Sangmin Bae Adam Fisch Hrayr Harutyunyan Ziwei Ji Seungyeon Kim Tal Schuster KELM 68 5 0 28 Oct 2024
Watermarking Large Language Models and the Generated Content: Opportunities and Challenges Ruisi Zhang F. Koushanfar WaLM 36 0 0 24 Oct 2024
Remote Timing Attacks on Efficient Language Model Inference Nicholas Carlini Milad Nasr 11 2 0 22 Oct 2024
MoDification: Mixture of Depths Made Easy C. Zhang M. Zhong Qimeng Wang Xuantao Lu Zheyu Ye ... Yan Gao Yao Hu Kehai Chen Min Zhang Dawei Song VLM MoE 35 2 0 18 Oct 2024
$γ-$ MoD: Exploring Mixture-of-Depth Adaptation for Multimodal Large Language Models Yaxin Luo Gen Luo Jiayi Ji Yiyi Zhou Xiaoshuai Sun Zhiqiang Shen Rongrong Ji VLM MoE 37 1 0 17 Oct 2024
big.LITTLE Vision Transformer for Efficient Visual Recognition He Guo Yulong Wang Zixuan Ye Jifeng Dai Yuwen Xiong ViT 50 0 0 14 Oct 2024
Skipping Computations in Multimodal LLMs Mustafa Shukor Matthieu Cord 24 2 0 12 Oct 2024
Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training Gen Luo Xue Yang Wenhan Dou Zhaokai Wang Jifeng Dai Jifeng Dai Yu Qiao Xizhou Zhu VLM MLLM 62 25 0 10 Oct 2024
A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models Cong Guo Feng Cheng Zhixu Du James Kiessling Jonathan Ku ... Qilin Zheng Guanglei Zhou Hai Li-Wei Li Yiran Chen 29 7 0 08 Oct 2024
SPikE-SSM: A Sparse, Precise, and Efficient Spiking State Space Model for Long Sequences Learning Yan Zhong Ruoyu Zhao Chao Wang Qinghai Guo Jianguo Zhang Zhichao Lu Luziwei Leng 44 2 0 07 Oct 2024
Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective Jinhao Li Jiaming Xu Shan Huang Yonghua Chen Wen Li ... Jiayi Pan Li Ding Hao Zhou Yu Wang Guohao Dai 57 15 0 06 Oct 2024
Duo-LLM: A Framework for Studying Adaptive Computation in Large Language Models Keivan Alizadeh Iman Mirzadeh Hooman Shahrokhi Dmitry Belenko Frank Sun Minsik Cho Mohammad Hossein Sekhavat Moin Nabi Mehrdad Farajtabar MoE 29 1 0 01 Oct 2024
CNN Mixture-of-Depths Rinor Cakaj Jens Mehnert Bin Yang 17 0 0 25 Sep 2024
VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computation Shiwei Wu Joya Chen Kevin Qinghong Lin Qimeng Wang Yan Gao Qianli Xu Tong Bill Xu Yao Hu Enhong Chen Mike Zheng Shou VLM 45 12 0 29 Aug 2024
MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Experts Xi Victoria Lin Akshat Shrivastava Liang Luo Srinivasan Iyer Mike Lewis Gargi Gosh Luke Zettlemoyer Armen Aghajanyan MoE 38 20 0 31 Jul 2024
Mixture of Nested Experts: Adaptive Processing of Visual Tokens Gagan Jain Nidhi Hegde Aditya Kusupati Arsha Nagrani Shyamal Buch Prateek Jain Anurag Arnab Sujoy Paul MoE 33 7 0 29 Jul 2024
A deeper look at depth pruning of LLMs Shoaib Ahmed Siddiqui Xin Dong Greg Heinrich Thomas Breuel Jan Kautz David M. Krueger Pavlo Molchanov 29 7 0 23 Jul 2024
Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules Zhuocheng Gong Ang Lv Jian-Yu Guan Junxi Yan Wei Yu Wu Huishuai Zhang Minlie Huang Dongyan Zhao Rui Yan MoE 44 6 0 09 Jul 2024
$$\text{Memory}^3$: Language Modeling with Explicit Memory$ $\text{Memory}^3$ : Language Modeling with Explicit Memory Hongkang Yang Zehao Lin Wenjin Wang Hao Wu Zhiyu Li ... Yu Yu Kai Chen Feiyu Xiong Linpeng Tang Weinan E 48 11 0 01 Jul 2024
Data curation via joint example selection further accelerates multimodal learning Talfan Evans Nikhil Parthasarathy Hamza Merzic Olivier J. Hénaff 32 12 0 25 Jun 2024
From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Models Sean Welleck Amanda Bertsch Matthew Finlayson Hailey Schoelkopf Alex Xie Graham Neubig Ilia Kulikov Zaid Harchaoui 33 47 0 24 Jun 2024
Block Transformer: Global-to-Local Language Modeling for Fast Inference Namgyu Ho Sangmin Bae Taehyeon Kim Hyunjik Jo Yireun Kim Tal Schuster Adam Fisch James Thorne Se-Young Yun 45 7 0 04 Jun 2024
Learning-to-Cache: Accelerating Diffusion Transformer via Layer Caching Xinyin Ma Gongfan Fang Michael Bi Mi Xinchao Wang 53 30 0 03 Jun 2024
SUBLLM: A Novel Efficient Architecture with Token Sequence Subsampling for LLM Quandong Wang Yuxuan Yuan Xiaoyu Yang Ruike Zhang Kang Zhao Wei Liu Jian Luan Daniel Povey Bin Wang 41 0 0 03 Jun 2024
Calibrating Reasoning in Language Models with Internal Consistency Zhihui Xie Jizhou Guo Tong Yu Shuai Li LRM 43 8 0 29 May 2024
Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Models Byung-Kwan Lee Chae Won Kim Beomchan Park Yonghyun Ro MLLM LRM 22 17 0 24 May 2024
MiniCache: KV Cache Compression in Depth Dimension for Large Language Models Akide Liu Jing Liu Zizheng Pan Yefei He Gholamreza Haffari Bohan Zhuang MQ 30 30 0 23 May 2024
Super Tiny Language Models Dylan Hillier Leon Guertler Cheston Tan Palaash Agrawal Ruirui Chen Bobby Cheng 45 3 0 23 May 2024
RMT-BVQA: Recurrent Memory Transformer-based Blind Video Quality Assessment for Enhanced Video Content Tianhao Peng Chen Feng Duolikun Danier Fan Zhang David Bull 28 2 0 14 May 2024
LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding Mostafa Elhoushi Akshat Shrivastava Diana Liskovich Basil Hosmer Bram Wasti ... Saurabh Agarwal Ahmed Roman Ahmed Aly Beidi Chen Carole-Jean Wu LRM 33 82 0 25 Apr 2024