Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing

5 June 2020

Papers citing "Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing"

50 / 141 papers shown

Title
Revisiting Funnel Transformers for Modern LLM Architectures with Comprehensive Ablations in Training and Inference Configurations DongHyun Choi Lucas Spangher Chris Hidey Peter Grabowski Ramy Eskander AI4CE 44 0 0 02 Apr 2025
Text Compression for Efficient Language Generation David Gu Peter Belcak Roger Wattenhofer 52 0 0 14 Mar 2025
Mamba-Shedder: Post-Transformer Compression for Efficient Selective Structured State Space Models J. P. Muñoz Jinjie Yuan Nilesh Jain Mamba 70 1 0 28 Jan 2025
Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMs Qizhe Zhang Aosong Cheng Ming Lu Zhiyong Zhuo Minqi Wang Jiajun Cao Shaobo Guo Qi She Shanghang Zhang VLM 90 11 0 02 Dec 2024
Can bidirectional encoder become the ultimate winner for downstream applications of foundation models? Lewen Yang Xuanyu Zhou Juao Fan Xinyi Xie Shengxin Zhu AI4CE 64 0 0 27 Nov 2024
Tiny Transformers Excel at Sentence Compression Peter Belcak Roger Wattenhofer 23 0 0 30 Oct 2024
Self-Satisfied: An end-to-end framework for SAT generation and prediction Christopher R. Serrano Jonathan Gallagher Kenji Yamada Alexei Kopylov Michael A. Warren 24 0 0 18 Oct 2024
FunnelRAG: A Coarse-to-Fine Progressive Retrieval Paradigm for RAG X. Zhao Yan Zhong Zetian Sun Xinshuo Hu Zhenyu Liu Dongfang Li Baotian Hu Min Zhang 58 6 0 14 Oct 2024
MegaFake: A Theory-Driven Dataset of Fake News Generated by Large Language Models Lionel Z. Wang Yiming Ma Renfei Gao Beichen Guo Han Zhu Wenqi Fan Zexin Lu Ka Chung Ng SyDa 23 2 0 19 Aug 2024
Finch: Prompt-guided Key-Value Cache Compression Giulio Corallo Paolo Papotti 38 3 0 31 Jul 2024
LLaVolta: Efficient Multi-modal Models via Stage-wise Visual Context Compression Jieneng Chen Luoxin Ye Ju He Zhao-Yang Wang Daniel Khashabi Alan Yuille VLM 27 5 0 28 Jun 2024
From Efficient Multimodal Models to World Models: A Survey Xinji Mai Zeng Tao Junxiong Lin Haoran Wang Yang Chang Yanlan Kang Yan Wang Wenqiang Zhang 32 5 0 27 Jun 2024
Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformers Chao Lou Zixia Jia Zilong Zheng Kewei Tu ODL 31 18 0 24 Jun 2024
Text Injection for Neural Contextual Biasing Zhong Meng Zelin Wu Rohit Prabhavalkar Cal Peyser Weiran Wang Nanxin Chen Tara N. Sainath Bhuvana Ramabhadran 20 3 0 05 Jun 2024
Block Transformer: Global-to-Local Language Modeling for Fast Inference Namgyu Ho Sangmin Bae Taehyeon Kim Hyunjik Jo Yireun Kim Tal Schuster Adam Fisch James Thorne Se-Young Yun 45 8 0 04 Jun 2024
SUBLLM: A Novel Efficient Architecture with Token Sequence Subsampling for LLM Quandong Wang Yuxuan Yuan Xiaoyu Yang Ruike Zhang Kang Zhao Wei Liu Jian Luan Daniel Povey Bin Wang 43 0 0 03 Jun 2024
Focus on the Core: Efficient Attention via Pruned Token Compression for Document Classification Jungmin Yun Mihyeon Kim Youngbin Kim 69 9 0 03 Jun 2024
Learning Robust Correlation with Foundation Model for Weakly-Supervised Few-Shot Segmentation Xinyang Huang Chuanglu Zhu Kebin Liu Ruiying Ren Shengjie Liu 33 2 0 30 May 2024
Activator: GLU Activation Function as the Core Component of a Vision Transformer Abdullah Nazhat Abdullah Tarkan Aydin ViT 38 0 0 24 May 2024
Efficient Multimodal Large Language Models: A Survey Yizhang Jin Jian Li Yexin Liu Tianjun Gu Kai Wu ... Xin Tan Zhenye Gan Yabiao Wang Chengjie Wang Lizhuang Ma LRM 41 45 0 17 May 2024
SpaceByte: Towards Deleting Tokenization from Large Language Modeling Kevin Slagle 32 3 0 22 Apr 2024
A Survey on Efficient Inference for Large Language Models Zixuan Zhou Xuefei Ning Ke Hong Tianyu Fu Jiaming Xu ... Shengen Yan Guohao Dai Xiao-Ping Zhang Yuhan Dong Yu-Xiang Wang 46 82 0 22 Apr 2024
Towards smaller, faster decoder-only transformers: Architectural variants and their implications Sathya Krishnan Suresh P. Shunmugapriya 19 0 0 22 Apr 2024
Explanation based Bias Decoupling Regularization for Natural Language Inference Jianxiang Zang Hui Liu 16 0 0 20 Apr 2024
Deferred NAM: Low-latency Top-K Context Injection via Deferred Context Encoding for Non-Streaming ASR Zelin Wu Gan Song Christopher Li Pat Rondon Zhong Meng ... D. Caseiro Golan Pundak Tsendsuren Munkhdalai Angad Chandorkar Rohit Prabhavalkar 18 3 0 15 Apr 2024
Learn to Code Sustainably: An Empirical Study on LLM-based Green Code Generation Tina Vartziotis Ippolyti Dellatolas George Dasoulas Maximilian Schmidt Florian Schneider Tim Hoffmann S. Kotsopoulos Michael Keckeisen 74 7 0 05 Mar 2024
A General and Flexible Multi-concept Parsing Framework for Multilingual Semantic Matching D. Yao Asaad Alghamdi Qingrong Xia Xiaoye Qu Xinyu Duan Zhefeng Wang Yi Zheng Baoxing Huai Peilun Cheng Zhou Zhao 24 0 0 05 Mar 2024
NiNformer: A Network in Network Transformer with Token Mixing Generated Gating Function Abdullah Nazhat Abdullah Tarkan Aydin 33 0 0 04 Mar 2024
Extreme Encoder Output Frame Rate Reduction: Improving Computational Latencies of Large End-to-End Models Rohit Prabhavalkar Zhong Meng Weiran Wang Adam Stooke Xingyu Cai Yanzhang He Arun Narayanan Dongseong Hwang Tara N. Sainath Pedro J. Moreno 25 8 0 27 Feb 2024
LLM Inference Unveiled: Survey and Roofline Model Insights Zhihang Yuan Yuzhang Shang Yang Zhou Zhen Dong Zhe Zhou ... Yong Jae Lee Yan Yan Beidi Chen Guangyu Sun Kurt Keutzer 37 79 0 26 Feb 2024
InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory Chaojun Xiao Pengle Zhang Xu Han Guangxuan Xiao Yankai Lin Zhengyan Zhang Zhiyuan Liu Maosong Sun LLMAG 39 34 0 07 Feb 2024
Sequence Shortening for Context-Aware Machine Translation Paweł Mąka Yusuf Can Semerci Jan Scholtes Gerasimos Spanakis 17 2 0 02 Feb 2024
BPDec: Unveiling the Potential of Masked Language Modeling Decoder in BERT pretraining Wen-Chieh Liang Youzhi Liang OffRL 23 2 0 29 Jan 2024
A Comprehensive Survey of Compression Algorithms for Language Models Seungcheol Park Jaehyeon Choi Sojin Lee U. Kang MQ 24 12 0 27 Jan 2024
MambaByte: Token-free Selective State Space Model Junxiong Wang Tushaar Gangavarapu Jing Nathan Yan Alexander M. Rush Mamba 36 34 0 24 Jan 2024
Flexibly Scaling Large Language Models Contexts Through Extensible Tokenization Ninglu Shao Shitao Xiao Zheng Liu Peitian Zhang 26 4 0 15 Jan 2024
TAROT: A Hierarchical Framework with Multitask Co-Pretraining on Semi-Structured Data towards Effective Person-Job Fit Yihan Cao Xu Chen Lun Du Hao Chen Qiang Fu Shi Han Yushu Du Yanbin Kang Guangming Lu Zi Li 33 0 0 15 Jan 2024
PEFA: Parameter-Free Adapters for Large-scale Embedding-based Retrieval Models Wei-Cheng Chang Jyun-Yu Jiang Jiong Zhang Mutasem Al-Darabsah C. Teo Cho-Jui Hsieh Hsiang-Fu Yu S. Vishwanathan RALM 19 2 0 05 Dec 2023
TPPoet: Transformer-Based Persian Poem Generation using Minimal Data and Advanced Decoding Techniques Amir Panahandeh Hanie Asemi Esmail Nourani 19 0 0 04 Dec 2023
Toucan: Token-Aware Character Level Language Modeling William Fleshman Benjamin Van Durme 17 3 0 15 Nov 2023
FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Cores Daniel Y. Fu Hermann Kumbong Eric N. D. Nguyen Christopher Ré VLM 36 29 0 10 Nov 2023
Recursion in Recursion: Two-Level Nested Recursion for Length Generalization with Scalability Jishnu Ray Chowdhury Cornelia Caragea 37 5 0 08 Nov 2023
Variator: Accelerating Pre-trained Models with Plug-and-Play Compression Modules Chaojun Xiao Yuqi Luo Wenbin Zhang Pengle Zhang Xu Han ... Zhengyan Zhang Ruobing Xie Zhiyuan Liu Maosong Sun Jie Zhou 22 0 0 24 Oct 2023
Low-Resource Clickbait Spoiling for Indonesian via Question Answering Ni Putu Intan Maharani Ayu Purwarianti Alham Fikri Aji 19 2 0 12 Oct 2023
Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs Suyu Ge Yunan Zhang Liyuan Liu Minjia Zhang Jiawei Han Jianfeng Gao 4 215 0 03 Oct 2023
Massive End-to-end Models for Short Search Queries Weiran Wang Rohit Prabhavalkar Dongseong Hwang Qiujia Li K. Sim ... Zhong Meng CJ Zheng Yanzhang He Tara N. Sainath P. M. Mengibar 22 2 0 22 Sep 2023
Towards Word-Level End-to-End Neural Speaker Diarization with Auxiliary Network Yiling Huang Weiran Wang Guanlong Zhao Hank Liao Wei Xia Quan Wang 17 4 0 15 Sep 2023
HM-Conformer: A Conformer-based audio deepfake detection system with hierarchical pooling and multi-level classification token aggregation methods Hyun-Seo Shin Ju-Sung Heo Ju-ho Kim Chanmann Lim Wonbin Kim Ha-Jin Yu 25 5 0 15 Sep 2023
A Comparative Analysis of Pretrained Language Models for Text-to-Speech M. G. Moya Panagiota Karanasou S. Karlapati Bastian Schnell Nicole Peinelt Alexis Moinet Thomas Drugman 37 3 0 04 Sep 2023
Enhancing Phenotype Recognition in Clinical Notes Using Large Language Models: PhenoBCBERT and PhenoGPT Jing Yang Cong Liu Wendy Deng Dangwei Wu C. Weng Yunyun Zhou Kai Wang 21 20 0 11 Aug 2023