Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice

9 March 2022

Papers citing "Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice"

50 / 74 papers shown

Title
Learning Advanced Self-Attention for Linear Transformers in the Singular Value Domain Hyowon Wi Jeongwhan Choi Noseong Park 20 0 0 13 May 2025
Cross Paradigm Representation and Alignment Transformer for Image Deraining Shun Zou Yi Zou J. Li Guangwei Gao Guojun Qi 35 0 0 23 Apr 2025
Spectral-Adaptive Modulation Networks for Visual Perception Guhnoo Yun J. Yoo Kijung Kim Jeongho Lee Paul Hongsuck Seo Dong Hwan Kim 37 0 0 31 Mar 2025
WaveFormer: A 3D Transformer with Wavelet-Driven Feature Representation for Efficient Medical Image Segmentation Md. Mahfuz Al Hasan Mahdi Zaman Abdul Jawad A. Santamaría-Pang Ho Hin Lee ... Md Shah Imran Antika Roy Y. P. Fallah Navid Asadizanjani Reza Forghani MedIm 37 0 0 31 Mar 2025
Hyperspectral Adapter for Object Tracking based on Hyperspectral Video Long Gao Yunhe Zhang Langkun Chen Yan Jiang Weiying Xie Yunsong Li VOS 53 0 0 28 Mar 2025
Coeff-Tuning: A Graph Filter Subspace View for Tuning Attention-Based Large Models Zichen Miao Wei Chen Qiang Qiu 90 1 0 24 Mar 2025
Mitigating Spectral Bias in Neural Operators via High-Frequency Scaling for Physical Systems Siavash Khodakarami Vivek Oommen Aniruddha Bora George Karniadakis AI4CE 60 1 0 17 Mar 2025
Transformer Meets Twicing: Harnessing Unattended Residual Information Laziz U. Abdullaev Tan M. Nguyen 39 2 0 02 Mar 2025
WiFi CSI Based Temporal Activity Detection via Dual Pyramid Network Zhendong Liu Le Zhang Bing Li Yingjie Zhou Z. Chen Ce Zhu 69 1 0 28 Jan 2025
MoRe: Class Patch Attention Needs Regularization for Weakly Supervised Semantic Segmentation Zhiwei Yang Yucong Meng Kexue Fu Shuo Wang Zhijian Song 85 1 0 20 Jan 2025
Boosting ViT-based MRI Reconstruction from the Perspectives of Frequency Modulation, Spatial Purification, and Scale Diversification Yucong Meng Zhiwei Yang Yonghong Shi Zhijian Song 77 2 0 14 Dec 2024
Unlocking Visual Secrets: Inverting Features with Diffusion Priors for Image Reconstruction Sai Qian Zhang Ziyun Li Chuan Guo Saeed Mahloujifar Deeksha Dangwal Edward Suh B. D. Salvo Chiao Liu DiffM 85 0 0 11 Dec 2024
Adaptive High-Pass Kernel Prediction for Efficient Video Deblurring Bo Ji Angela Yao 73 0 0 02 Dec 2024
Enhancing Parameter-Efficient Fine-Tuning of Vision Transformers through Frequency-Based Adaptation S. Ly Hien Nguyen 72 1 0 28 Nov 2024
GrokFormer: Graph Fourier Kolmogorov-Arnold Transformers Guoguo Ai Guansong Pang Hezhe Qiao Yuan Gao Hui Yan 67 0 0 26 Nov 2024
Length-Induced Embedding Collapse in Transformer-based Models Yuqi Zhou Sunhao Dai Zhanshuo Cao Xiao Zhang Jun Xu 37 0 0 31 Oct 2024
Generalized Probabilistic Attention Mechanism in Transformers DongNyeong Heo Heeyoul Choi 49 0 0 21 Oct 2024
CATCH: Channel-Aware multivariate Time Series Anomaly Detection via Frequency Patching Xingjian Wu Xiangfei Qiu Zhengyu Li Yihang Wang Jilin Hu Chenjuan Guo Hui Xiong Bin Yang AI4TS 59 12 0 16 Oct 2024
Spiking Transformer with Spatial-Temporal Attention Donghyun Lee Yuhang Li Youngeun Kim Shiting Xiao Priyadarshini Panda 26 1 0 29 Sep 2024
Down-Sampling Inter-Layer Adapter for Parameter and Computation Efficient Ultra-Fine-Grained Image Recognition Edwin Arkel Rios Femiloye Oyerinde Min-Chun Hu Bo-Cheng Lai 24 0 0 17 Sep 2024
Hybrid-Segmentor: A Hybrid Approach to Automated Fine-Grained Crack Segmentation in Civil Infrastructure June Moh Goo Xenios Milidonis Alessandro Artusi Jan Boehm Carlo Ciliberto 25 4 0 04 Sep 2024
MSA $^2$ Net: Multi-scale Adaptive Attention-guided Network for Medical Image Segmentation Sina Ghorbani Kolahi S. K. Chaharsooghi Toktam Khatibi Afshin Bozorgpour Reza Azad Moein Heidari I. Hacihaliloglu Dorit Merhof 24 1 0 31 Jul 2024
Attention Is All You Need But You Don't Need All Of It For Inference of Large Language Models Georgy Tyukin G. Dovonon Jean Kaddour Pasquale Minervini LRM 31 0 0 22 Jul 2024
Foundation Models for Autonomous Robots in Unstructured Environments Hossein Naderi Alireza Shojaei Lifu Huang LM&Ro 43 0 0 19 Jul 2024
Adaptive Parametric Activation Konstantinos Panagiotis Alexandridis Jiankang Deng Anh Nguyen Shan Luo 36 2 0 11 Jul 2024
Unveiling the Hidden Structure of Self-Attention via Kernel Principal Component Analysis R. Teo Tan M. Nguyen 43 4 0 19 Jun 2024
Fredformer: Frequency Debiased Transformer for Time Series Forecasting Xihao Piao Zheng Chen Taichi Murayama Yasuko Matsubara Yasushi Sakurai AI4TS 27 15 0 13 Jun 2024
MultiMax: Sparse and Multi-Modal Attention Learning Yuxuan Zhou Mario Fritz M. Keuper 40 1 0 03 Jun 2024
Theoretical Bound-Guided Hierarchical VAE for Neural Image Codecs Yichi Zhang Zhihao Duan Yuning Huang F. Zhu 16 4 0 27 Mar 2024
Attacking Transformers with Feature Diversity Adversarial Perturbation Chenxing Gao Hang Zhou Junqing Yu Yuteng Ye Jiale Cai Junle Wang Wei Yang AAML 32 3 0 10 Mar 2024
The Hidden Attention of Mamba Models Ameen Ali Itamar Zimerman Lior Wolf Mamba 37 58 0 03 Mar 2024
PreRoutGNN for Timing Prediction with Order Preserving Partition: Global Circuit Pre-training, Local Delay Learning and Attentional Cell Modeling Ruizhe Zhong Junjie Ye Zhentao Tang Shixiong Kai Mingxuan Yuan Jianye Hao Junchi Yan 34 8 0 27 Feb 2024
PIDformer: Transformer Meets Control Theory Tam Nguyen César A. Uribe Tan-Minh Nguyen Richard G. Baraniuk 48 7 0 25 Feb 2024
Setting the Record Straight on Transformer Oversmoothing G. Dovonon M. Bronstein Matt J. Kusner 20 5 0 09 Jan 2024
A mathematical perspective on Transformers Borjan Geshkovski Cyril Letrouit Yury Polyanskiy Philippe Rigollet EDL AI4CE 40 36 0 17 Dec 2023
An Attentive Inductive Bias for Sequential Recommendation beyond the Self-Attention Yehjin Shin Jeongwhan Choi Hyowon Wi Noseong Park 38 29 0 16 Dec 2023
Polynomial-based Self-Attention for Table Representation learning Jayoung Kim Yehjin Shin Jeongwhan Choi Hyowon Wi Noseong Park LMTD 19 2 0 12 Dec 2023
Graph Convolutions Enrich the Self-Attention in Transformers! Jeongwhan Choi Hyowon Wi Jayoung Kim Yehjin Shin Kookjin Lee Nathaniel Trask Noseong Park 25 4 0 07 Dec 2023
Mitigating Over-smoothing in Transformers via Regularized Nonlocal Functionals Tam Nguyen Tan-Minh Nguyen Richard G. Baraniuk 21 8 0 01 Dec 2023
p-Laplacian Transformer Tuan Nguyen Tam Nguyen Vinh-Tiep Nguyen Tan-Minh Nguyen 69 0 0 06 Nov 2023
Simplifying Transformer Blocks Bobby He Thomas Hofmann 19 30 0 03 Nov 2023
Foundational Models in Medical Imaging: A Comprehensive Survey and Future Vision Bobby Azad Reza Azad Sania Eskandari Afshin Bozorgpour A. Kazerouni I. Rekik Dorit Merhof VLM MedIm 93 59 0 28 Oct 2023
PartialFormer: Modeling Part Instead of Whole for Machine Translation Tong Zheng Bei Li Huiwen Bao Jiale Wang Weiqiao Shan Tong Xiao Jingbo Zhu MoE AI4CE 11 0 0 23 Oct 2023
Inter-Scale Dependency Modeling for Skin Lesion Segmentation with Transformer-based Networks Sania Eskandari Janet Lumpp ViT MedIm 13 3 0 20 Oct 2023
Are GATs Out of Balance? Nimrah Mustafa Aleksandar Bojchevski R. Burkholz 46 4 0 11 Oct 2023
No Token Left Behind: Efficient Vision Transformer via Dynamic Token Idling Xuwei Xu Changlin Li Yudong Chen Xiaojun Chang Jiajun Liu Sen Wang ViT 18 5 0 09 Oct 2023
Hydra: Multi-head Low-rank Adaptation for Parameter Efficient Fine-tuning Sanghyeon Kim Hyunmo Yang Younghyun Kim Youngjoon Hong Eunbyung Park AI4CE 18 16 0 13 Sep 2023
Beyond Self-Attention: Deformable Large Kernel Attention for Medical Image Segmentation Reza Azad Leon Niggemeier Michael Huttemann A. Kazerouni Ehsan Khodapanah Aghdam Yury Velichko Ulas Bagci Dorit Merhof MedIm 22 55 0 31 Aug 2023
Laplacian-Former: Overcoming the Limitations of Vision Transformers in Local Texture Detection Reza Azad A. Kazerouni Babak Azad Ehsan Khodapanah Aghdam Yury Velichko Ulas Bagci Dorit Merhof ViT MedIm 11 16 0 31 Aug 2023
Unlocking Fine-Grained Details with Wavelet-based High-Frequency Enhancement in Transformers Reza Azad A. Kazerouni Alaa Sulaiman Afshin Bozorgpour Ehsan Khodapanah Aghdam Abin Jose Dorit Merhof ViT MedIm 16 9 0 25 Aug 2023