Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice

International Conference on Learning Representations (ICLR), 2022

9 March 2022

ArXiv (abs)PDF HTML Github (79★)

Papers citing "Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice"

50 / 100 papers shown

Title
Unveiling the Hidden Structure of Self-Attention via Kernel Principal Component Analysis R. Teo Tan M. Nguyen 297 7 0 19 Jun 2024
Fredformer: Frequency Debiased Transformer for Time Series Forecasting Xihao Piao Zheng Chen Taichi Murayama Yasuko Matsubara Yasushi Sakurai AI4TS 243 62 0 13 Jun 2024
MultiMax: Sparse and Multi-Modal Attention Learning Yuxuan Zhou Mario Fritz Margret Keuper 369 1 0 03 Jun 2024
Frequency Decomposition-Driven Unsupervised Domain Adaptation for Remote Sensing Image Semantic Segmentation Xianping Ma Xiaokang Zhang Xingchen Ding Man-On Pun Siwei Ma 126 2 0 06 Apr 2024
Theoretical Bound-Guided Hierarchical VAE for Neural Image Codecs Yichi Zhang Zhihao Duan Yuning Huang Fengqing Zhu 118 6 0 27 Mar 2024
Attacking Transformers with Feature Diversity Adversarial PerturbationAAAI Conference on Artificial Intelligence (AAAI), 2024 Chenxing Gao Hang Zhou Junqing Yu Yuteng Ye Jiale Cai Junle Wang Wei Yang AAML 166 4 0 10 Mar 2024
The Hidden Attention of Mamba Models Ameen Ali Itamar Zimerman Lior Wolf Mamba 401 89 0 03 Mar 2024
PreRoutGNN for Timing Prediction with Order Preserving Partition: Global Circuit Pre-training, Local Delay Learning and Attentional Cell Modeling Ruizhe Zhong Junjie Ye Zhentao Tang Shixiong Kai Mingxuan Yuan Jianye Hao Junchi Yan 144 16 0 27 Feb 2024
PIDformer: Transformer Meets Control Theory Tam Nguyen César A. Uribe Tan-Minh Nguyen Richard G. Baraniuk 231 10 0 25 Feb 2024
Setting the Record Straight on Transformer Oversmoothing G. Dovonon M. Bronstein Matt J. Kusner 316 11 0 09 Jan 2024
A mathematical perspective on TransformersBulletin of the American Mathematical Society (BAMS), 2023 Borjan Geshkovski Cyril Letrouit Yury Polyanskiy Philippe Rigollet EDL AI4CE 540 95 0 17 Dec 2023
An Attentive Inductive Bias for Sequential Recommendation beyond the Self-AttentionAAAI Conference on Artificial Intelligence (AAAI), 2023 Yehjin Shin Jeongwhan Choi Hyowon Wi Noseong Park 208 90 0 16 Dec 2023
Polynomial-based Self-Attention for Table Representation learningInternational Conference on Machine Learning (ICML), 2023 Jayoung Kim Yehjin Shin Jeongwhan Choi Hyowon Wi Noseong Park LMTD 136 3 0 12 Dec 2023
Graph Convolutions Enrich the Self-Attention in Transformers! Jeongwhan Choi Hyowon Wi Jayoung Kim Yehjin Shin Kookjin Lee Nathaniel Trask Noseong Park 310 12 0 07 Dec 2023
Mitigating Over-smoothing in Transformers via Regularized Nonlocal FunctionalsNeural Information Processing Systems (NeurIPS), 2023 Tam Nguyen Tan-Minh Nguyen Richard G. Baraniuk 154 25 0 01 Dec 2023
p-Laplacian Transformer Tuan Nguyen Tam Nguyen Vinh-Tiep Nguyen Tan-Minh Nguyen 166 0 0 06 Nov 2023
Simplifying Transformer BlocksInternational Conference on Learning Representations (ICLR), 2023 Bobby He Thomas Hofmann 312 45 0 03 Nov 2023
Foundational Models in Medical Imaging: A Comprehensive Survey and Future Vision Bobby Azad Reza Azad Sania Eskandari Afshin Bozorgpour Amirhossein Kazerouni I. Rekik Dorit Merhof VLM MedIm 214 98 0 28 Oct 2023
PartialFormer: Modeling Part Instead of Whole for Machine TranslationAnnual Meeting of the Association for Computational Linguistics (ACL), 2023 Tong Zheng Bei Li Huiwen Bao Jiale Wang Weiqiao Shan Tong Xiao Jingbo Zhu MoE AI4CE 190 1 0 23 Oct 2023
Inter-Scale Dependency Modeling for Skin Lesion Segmentation with Transformer-based Networks Sania Eskandari Janet Lumpp ViT MedIm 74 5 0 20 Oct 2023
Are GATs Out of Balance?Neural Information Processing Systems (NeurIPS), 2023 Nimrah Mustafa Aleksandar Bojchevski R. Burkholz 294 8 0 11 Oct 2023
No Token Left Behind: Efficient Vision Transformer via Dynamic Token IdlingApplied Informatics (AI), 2023 Xuwei Xu Changlin Li Yudong Chen Xiaojun Chang Jiajun Liu Sen Wang ViT 186 9 0 09 Oct 2023
Hydra: Multi-head Low-rank Adaptation for Parameter Efficient Fine-tuningNeural Networks (Neural Netw.), 2023 Sanghyeon Kim Hyunmo Yang Younghyun Kim Youngjoon Hong Eunbyung Park AI4CE 191 30 0 13 Sep 2023
Beyond Self-Attention: Deformable Large Kernel Attention for Medical Image SegmentationIEEE Workshop/Winter Conference on Applications of Computer Vision (WACV), 2023 Reza Azad Leon Niggemeier Michael Huttemann Amirhossein Kazerouni Ehsan Khodapanah Aghdam Yury Velichko Ulas Bagci Dorit Merhof MedIm 240 113 0 31 Aug 2023
Laplacian-Former: Overcoming the Limitations of Vision Transformers in Local Texture DetectionInternational Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2023 Reza Azad Amirhossein Kazerouni Babak Azad Ehsan Khodapanah Aghdam Yury Velichko Ulas Bagci Dorit Merhof ViT MedIm 208 25 0 31 Aug 2023
Unlocking Fine-Grained Details with Wavelet-based High-Frequency Enhancement in Transformers Reza Azad Amirhossein Kazerouni Alaa Sulaiman Afshin Bozorgpour Ehsan Khodapanah Aghdam Abin Jose Dorit Merhof ViT MedIm 158 16 0 25 Aug 2023
Diverse Cotraining Makes Strong Semi-Supervised SegmentorIEEE International Conference on Computer Vision (ICCV), 2023 Yijiang Li Xinjiang Wang Lihe Yang Xue Jiang Wayne Zhang Ying Gao 154 34 0 18 Aug 2023
Revisiting Vision Transformer from the View of Path EnsembleIEEE International Conference on Computer Vision (ICCV), 2023 Shuning Chang Pichao Wang Haowen Luo Fan Wang Mike Zheng Shou ViT 122 7 0 12 Aug 2023
Towards Building More Robust Models with Frequency BiasIEEE International Conference on Computer Vision (ICCV), 2023 Qingwen Bu Dong Huang Heming Cui AAML 185 18 0 19 Jul 2023
3DSAM-adapter: Holistic Adaptation of SAM from 2D to 3D for Promptable Medical Image Segmentation Shizhan Gong Yuan Zhong Wenao Ma Jinpeng Li Zhao Wang Jingyang Zhang Pheng-Ann Heng Qi Dou MedIm 117 60 0 23 Jun 2023
Multi-Architecture Multi-Expert Diffusion ModelsAAAI Conference on Artificial Intelligence (AAAI), 2023 Yunsung Lee Jin-Young Kim Hyojun Go Myeongho Jeong Shinhyeok Oh Seungtaek Choi DiffM 298 37 0 08 Jun 2023
Centered Self-Attention LayersScandinavian Conference on Image Analysis (SCIA), 2023 Ameen Ali Tomer Galanti Lior Wolf 380 8 0 02 Jun 2023
SmartTrim: Adaptive Tokens and Attention Pruning for Efficient Vision-Language ModelsInternational Conference on Language Resources and Evaluation (LREC), 2023 Zekun Wang Jingchang Chen Wangchunshu Zhou Haichao Zhu Jiafeng Liang Liping Shan Ming Liu Dongliang Xu Qing Yang Bing Qin VLM 190 8 0 24 May 2023
ScatterFormer: Locally-Invariant Scattering Transformer for Patient-Independent Multispectral Detection of Epileptiform DischargesAAAI Conference on Artificial Intelligence (AAAI), 2023 Rui-Hua Zheng Jun Yu Li Yi Wang Tian Luo Yuguo Yu MedIm 171 8 0 26 Apr 2023
SRFormer: Permuted Self-Attention for Single Image Super-Resolution Yupeng Zhou Zerui Li Chunle Guo S. Bai Ming-Ming Cheng Qibin Hou 92 37 0 17 Mar 2023
Token Contrast for Weakly-Supervised Semantic SegmentationComputer Vision and Pattern Recognition (CVPR), 2023 Lixiang Ru Heliang Zheng Yibing Zhan Bo Du ViT 288 131 0 02 Mar 2023
Specformer: Spectral Graph Neural Networks Meet TransformersInternational Conference on Learning Representations (ICLR), 2023 Deyu Bo Chuan Shi Lele Wang Renjie Liao 239 123 0 02 Mar 2023
Are More Layers Beneficial to Graph Transformers?International Conference on Learning Representations (ICLR), 2023 Haiteng Zhao Shuming Ma Dongdong Zhang Zhi-Hong Deng Furu Wei 165 17 0 01 Mar 2023
EIT: Enhanced Interactive TransformerAnnual Meeting of the Association for Computational Linguistics (ACL), 2022 Tong Zheng Bei Li Huiwen Bao Tong Xiao Jingbo Zhu 270 3 0 20 Dec 2022
MogaNet: Multi-order Gated Aggregation NetworkInternational Conference on Learning Representations (ICLR), 2022 Siyuan Li Zedong Wang Zicheng Liu Cheng Tan Haitao Lin Di Wu Zhiyuan Chen Jiangbin Zheng Stan Z. Li 245 112 0 07 Nov 2022
M $^3$ ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designNeural Information Processing Systems (NeurIPS), 2022 Hanxue Liang Zhiwen Fan Rishov Sarkar Ziyu Jiang Tianlong Chen Kai Zou Yu Cheng Cong Hao Zinan Lin MoE 186 123 0 26 Oct 2022
Old can be Gold: Better Gradient Flow can Make Vanilla-GCNs Great AgainNeural Information Processing Systems (NeurIPS), 2022 Ajay Jaiswal Peihao Wang Tianlong Chen Justin F. Rousseau Ying Ding Zinan Lin 144 15 0 14 Oct 2022
Can We Solve 3D Vision Tasks Starting from A 2D Vision Transformer? Yi Wang Zhiwen Fan Tianlong Chen Hehe Fan Zinan Lin ViT 209 10 0 15 Sep 2022
Learning Spatial-Frequency Transformer for Visual Object Tracking Ju Huang Tianlin Li Yuanchao Bai Zhe Wu Jianlin Zhang Yongmei Huang ViT 300 65 0 18 Aug 2022
A Study on Transformer Configuration and Training ObjectiveInternational Conference on Machine Learning (ICML), 2022 Fuzhao Xue Jianghai Chen Aixin Sun Xiaozhe Ren Zangwei Zheng Xiaoxin He Yongming Chen Xin Jiang Yang You 138 10 0 21 May 2022
Improving Vision Transformers by Revisiting High-frequency ComponentsEuropean Conference on Computer Vision (ECCV), 2022 Jiawang Bai Liuliang Yuan Shutao Xia Shuicheng Yan Zhifeng Li Wen Liu ViT 232 115 0 03 Apr 2022
What Do Adversarially trained Neural Networks Focus: A Fourier Domain-based Study Binxiao Huang Chaofan Tao R. Lin Ngai Wong AAML OOD 130 3 0 16 Mar 2022
Symbolic Learning to Optimize: Towards Interpretability and ScalabilityInternational Conference on Learning Representations (ICLR), 2022 Wenqing Zheng Tianlong Chen Ting-Kuei Hu Zinan Lin 357 21 0 13 Mar 2022
Bag of Tricks for Training Deeper Graph Neural Networks: A Comprehensive Benchmark StudyIEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2021 Tianlong Chen Kaixiong Zhou Keyu Duan Wenqing Zheng Peihao Wang Helen Zhou Zinan Lin AAML GNN 117 73 0 24 Aug 2021
GeoT: A Geometry-aware Transformer for Reliable Molecular Property Prediction and Chemically Interpretable Representation LearningACS Omega (ACS Omega), 2021 Bumju Kwak J. Park Taewon Kang Jeonghee Jo Byunghan Lee Sungroh Yoon AI4CE 161 6 0 29 Jun 2021