FNet: Mixing Tokens with Fourier Transforms

9 May 2021

James Lee-Thorp

Joshua Ainslie

Ilya Eckstein

Santiago Ontanon

ArXiv PDF HTML

Papers citing "FNet: Mixing Tokens with Fourier Transforms"

50 / 251 papers shown

Title
Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challenges Badri N. Patro Vijay Srinivas Agneeswaran Mamba 46 38 0 24 Apr 2024
MTKD: Multi-Teacher Knowledge Distillation for Image Super-Resolution Yuxuan Jiang Chen Feng Fan Zhang David Bull SupR 51 11 0 15 Apr 2024
Softmax Attention with Constant Cost per Token Franz A. Heinsen 29 1 0 08 Apr 2024
Model Selection with Model Zoo via Graph Learning Ziyu Li Hilco van der Wilk Danning Zhan Megha Khosla A. Bozzon Rihan Hai 46 1 0 05 Apr 2024
SpiralMLP: A Lightweight Vision MLP Architecture Haojie Mu Burhan Ul Tayyab Nicholas Chua 43 0 0 31 Mar 2024
DiJiang: Efficient Large Language Models through Compact Kernelization Hanting Chen Zhicheng Liu Xutao Wang Yuchuan Tian Yunhe Wang VLM 31 5 0 29 Mar 2024
Fourier or Wavelet bases as counterpart self-attention in spikformer for efficient visual classification Qingyu Wang Duzhen Zhang Tilelin Zhang Bo Xu 39 1 0 27 Mar 2024
Heracles: A Hybrid SSM-Transformer Model for High-Resolution Image and Time-Series Analysis Badri N. Patro Suhas Ranganath Vinay P. Namboodiri Vijay Srinivas Agneeswaran 43 2 0 26 Mar 2024
Holographic Global Convolutional Networks for Long-Range Prediction Tasks in Malware Detection Mohammad Mahmudul Alam Edward Raff Stella Biderman Tim Oates James Holt AAML 30 3 0 23 Mar 2024
SiMBA: Simplified Mamba-Based Architecture for Vision and Multivariate Time series Badri N. Patro Vijay Srinivas Agneeswaran Mamba 59 50 0 22 Mar 2024
Learning with SASQuaTCh: a Novel Variational Quantum Transformer Architecture with Kernel-Based Self-Attention Ethan N. Evans Matthew G. Cook Zachary P. Bradshaw Margarite L. LaBorde 48 5 0 21 Mar 2024
Adaptive Ensembles of Fine-Tuned Transformers for LLM-Generated Text Detection Zhixin Lai Xuesheng Zhang Suiyao Chen DeLMO 41 30 0 20 Mar 2024
Deep adaptative spectral zoom for improved remote heart rate estimation Joaquim Comas Adria Ruiz Federico Sukno 52 1 0 11 Mar 2024
Not just Birds and Cars: Generic, Scalable and Explainable Models for Professional Visual Recognition Junde Wu Jiayuan Zhu Min Xu Yueming Jin 35 0 0 08 Mar 2024
Training Machine Learning models at the Edge: A Survey Aymen Rayane Khouas Mohamed Reda Bouadjenek Hakim Hacid Sunil Aryal 29 10 0 05 Mar 2024
NiNformer: A Network in Network Transformer with Token Mixing Generated Gating Function Abdullah Nazhat Abdullah Tarkan Aydin 39 0 0 04 Mar 2024
HyenaPixel: Global Image Context with Convolutions Julian Spravil Sebastian Houben Sven Behnke 31 1 0 29 Feb 2024
Windowed-FourierMixer: Enhancing Clutter-Free Room Modeling with Fourier Transform Bruno Henriques Benjamin Allaert Jean-Philippe Vandeborre 3DV 32 0 0 28 Feb 2024
Learning to See Through Dazzle Xiaopeng Peng Erin F. Fleet A. Watnik Grover A. Swartzlander GAN AAML 32 4 0 24 Feb 2024
FViT: A Focal Vision Transformer with Gabor Filter Yulong Shi Mingwei Sun Yongshuai Wang Rui Wang 60 4 0 17 Feb 2024
CAST: Clustering Self-Attention using Surrogate Tokens for Efficient Transformers Adjorn van Engelenhoven Nicola Strisciuglio Estefanía Talavera 23 1 0 06 Feb 2024
LOCOST: State-Space Models for Long Document Abstractive Summarization Florian Le Bronnec Song Duong Mathieu Ravaut Alexandre Allauzen Nancy F. Chen Vincent Guigue Alberto Lumbreras Laure Soulier Patrick Gallinari 48 8 0 31 Jan 2024
Fourier Prompt Tuning for Modality-Incomplete Scene Segmentation Ruiping Liu Jiaming Zhang Kunyu Peng Yufan Chen Ke Cao Junwei Zheng M. Sarfraz Kailun Yang Rainer Stiefelhagen VLM 42 8 0 30 Jan 2024
Transformers are Multi-State RNNs Matanel Oren Michael Hassid Nir Yarden Yossi Adi Roy Schwartz OffRL 32 35 0 11 Jan 2024
AugSumm: towards generalizable speech summarization using synthetic labels from large language model Jee-weon Jung Roshan S. Sharma William Chen Bhiksha Raj Shinji Watanabe 53 4 0 10 Jan 2024
Heterogeneous Encoders Scaling In The Transformer For Neural Machine Translation J. Hu Roberto Cavicchioli Giulia Berardinelli Alessandro Capotondi 44 2 0 26 Dec 2023
Efficiency-oriented approaches for self-supervised speech representation learning Luis Lugo Valentin Vielzeuf SSL 29 1 0 18 Dec 2023
Linear Attention via Orthogonal Memory Jun Zhang Shuyang Jiang Jiangtao Feng Lin Zheng Lingpeng Kong 40 3 0 18 Dec 2023
Frequency Spectrum is More Effective for Multimodal Representation and Fusion: A Multimodal Spectrum Rumor Detector An Lao Qi Zhang Chongyang Shi LongBing Cao Kun Yi Liang Hu Duoqian Miao 44 20 0 18 Dec 2023
Learning Long Sequences in Spiking Neural Networks Matei Ioan Stan Oliver Rhodes 37 11 0 14 Dec 2023
Graph Convolutions Enrich the Self-Attention in Transformers! Jeongwhan Choi Hyowon Wi Jayoung Kim Yehjin Shin Kookjin Lee Nathaniel Trask Noseong Park 32 4 0 07 Dec 2023
Bootstrapping Interactive Image-Text Alignment for Remote Sensing Image Captioning Cong Yang Zuchao Li Lefei Zhang 31 23 0 02 Dec 2023
Improving Interpretation Faithfulness for Vision Transformers Lijie Hu Yixin Liu Ninghao Liu Mengdi Huai Lichao Sun Di Wang 41 5 0 29 Nov 2023
Do large language models and humans have similar behaviors in causal inference with script knowledge? Xudong Hong Margarita Ryzhova Daniel Adrian Biondi Ram Sarkar 42 5 0 13 Nov 2023
Legal-HNet: Mixing Legal Long-Context Tokens with Hartley Transform Daniele Giofré Sneha Ghantasala AILaw 29 0 0 09 Nov 2023
Hierarchically Gated Recurrent Neural Network for Sequence Modeling Zhen Qin Aaron Courville Yiran Zhong 36 74 0 08 Nov 2023
Scattering Vision Transformer: Spectral Mixing Matters Badri N. Patro Vijay Srinivas Agneeswaran 39 14 0 02 Nov 2023
How well can machine-generated texts be identified and can language models be trained to avoid identification? Sinclair Schneider Florian Steuber João A. G. Schneider Gabi Dreo Rodosek DeLMO 28 1 0 25 Oct 2023
Pit One Against Many: Leveraging Attention-head Embeddings for Parameter-efficient Multi-head Attention Huiyin Xue Nikolaos Aletras 30 0 0 11 Oct 2023
Distance Weighted Trans Network for Image Completion Pourya Shamsolmoali Masoumeh Zareapoor Huiyu Zhou Xuelong Li Yue Lu ViT 33 0 0 11 Oct 2023
Exponential Quantum Communication Advantage in Distributed Inference and Learning H. Michaeli D. Gilboa Daniel Soudry Jarrod R. McClean FedML 19 0 0 11 Oct 2023
CrysFormer: Protein Structure Prediction via 3d Patterson Maps and Partial Structure Attention Chen Dun Qiutai Pan Shikai Jin Ria Stevens Mitchell D. Miller George N. Phillips Anastasios Kyrillidis 12 2 0 05 Oct 2023
HartleyMHA: Self-Attention in Frequency Domain for Resolution-Robust and Parameter-Efficient 3D Image Segmentation Ken C. L. Wong Hongzhi Wang T. Syeda-Mahmood 42 1 0 05 Oct 2023
Transformer-VQ: Linear-Time Transformers via Vector Quantization Albert Mohwald 31 15 0 28 Sep 2023
$Only 5\% Attention Is All You Need: Efficient Long-range Document-level Neural Machine Translation$ Only 5\% Attention Is All You Need: Efficient Long-range Document-level Neural Machine Translation Zihan Liu Zewei Sun Shanbo Cheng Shujian Huang Mingxuan Wang 28 1 0 25 Sep 2023
Generative AI vs. AGI: The Cognitive Strengths and Weaknesses of Modern LLMs Ben Goertzel 38 13 0 19 Sep 2023
Dynamic Spectrum Mixer for Visual Recognition Zhiqiang Hu Tao Yu 30 3 0 13 Sep 2023
SPANet: Frequency-balancing Token Mixer using Spectral Pooling Aggregation Modulation Guhnoo Yun J. Yoo Kijung Kim Jeongho Lee Dong Hwan Kim MoE 33 8 0 22 Aug 2023
Which Transformer to Favor: A Comparative Analysis of Efficiency in Vision Transformers Tobias Christian Nauen Sebastián M. Palacio Federico Raue Andreas Dengel 42 3 0 18 Aug 2023
Attention Is Not All You Need Anymore Zhe Chen 32 3 0 15 Aug 2023