On the Relationship between Self-Attention and Convolutional Layers

8 November 2019

Papers citing "On the Relationship between Self-Attention and Convolutional Layers"

46 / 96 papers shown

Title
Couplformer:Rethinking Vision Transformer with Coupling Attention Map Hai Lan Xihao Wang Xian Wei ViT 26 3 0 10 Dec 2021
On the Integration of Self-Attention and Convolution Xuran Pan Chunjiang Ge Rui Lu S. Song Guanfu Chen Zeyi Huang Gao Huang SSL 36 287 0 29 Nov 2021
Multi-domain Integrative Swin Transformer network for Sparse-View Tomographic Reconstruction Jiayi Pan Heye Zhang Weifei Wu Z. Gao Weiwen Wu 16 59 0 28 Nov 2021
Pruning Self-attentions into Convolutional Layers in Single Path Haoyu He Jianfei Cai Jing Liu Zizheng Pan Jing Zhang Dacheng Tao Bohan Zhuang ViT 31 40 0 23 Nov 2021
Full-attention based Neural Architecture Search using Context Auto-regression Yuan Zhou Haiyang Wang Shuwei Huo Boyu Wang 25 3 0 13 Nov 2021
A Survey of Visual Transformers Yang Liu Yao Zhang Yixin Wang Feng Hou Jin Yuan Jiang Tian Yang Zhang Zhongchao Shi Jianping Fan Zhiqiang He 3DGS ViT 71 330 0 11 Nov 2021
Can Vision Transformers Perform Convolution? Shanda Li Xiangning Chen Di He Cho-Jui Hsieh ViT 35 19 0 02 Nov 2021
ASFormer: Transformer for Action Segmentation Fangqiu Yi Hongyu Wen Tingting Jiang ViT 73 172 0 16 Oct 2021
FlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizes David W. Romero Robert-Jan Bruintjes Jakub M. Tomczak Erik J. Bekkers Mark Hoogendoorn J. C. V. Gemert 80 81 0 15 Oct 2021
Dynamic Inference with Neural Interpreters Nasim Rahaman Muhammad Waleed Gondal S. Joshi Peter V. Gehler Yoshua Bengio Francesco Locatello Bernhard Schölkopf 34 31 0 12 Oct 2021
Learning the Physics of Particle Transport via Transformers O. Pastor-Serrano Zoltán Perkó MedIm 21 13 0 08 Sep 2021
SwinIR: Image Restoration Using Swin Transformer Jingyun Liang Jie Cao Guolei Sun K. Zhang Luc Van Gool Radu Timofte ViT 42 2,806 0 23 Aug 2021
Do Vision Transformers See Like Convolutional Neural Networks? M. Raghu Thomas Unterthiner Simon Kornblith Chiyuan Zhang Alexey Dosovitskiy ViT 46 924 0 19 Aug 2021
RaftMLP: How Much Can Be Done Without Attention and with Less Spatial Locality? Yuki Tatsunami Masato Taki 24 12 0 09 Aug 2021
Global Self-Attention as a Replacement for Graph Convolution Md Shamim Hussain Mohammed J. Zaki D. Subramanian ViT 31 121 0 07 Aug 2021
CycleMLP: A MLP-like Architecture for Dense Prediction Shoufa Chen Enze Xie Chongjian Ge Runjian Chen Ding Liang Ping Luo 19 231 0 21 Jul 2021
Transformer with Peak Suppression and Knowledge Guidance for Fine-grained Image Recognition Xinda Liu Lili Wang Xiaoguang Han ViT 34 66 0 14 Jul 2021
Visual Parser: Representing Part-whole Hierarchies with Transformers Shuyang Sun Xiaoyu Yue S. Bai Philip H. S. Torr 50 27 0 13 Jul 2021
Can Transformers Jump Around Right in Natural Language? Assessing Performance Transfer from SCAN Rahma Chaabouni Roberto Dessì Eugene Kharitonov 19 20 0 03 Jul 2021
Polarized Self-Attention: Towards High-quality Pixel-wise Regression Huajun Liu Fuqiang Liu Xinyi Fan Dong Huang 72 211 0 02 Jul 2021
Co-advise: Cross Inductive Bias Distillation Sucheng Ren Zhengqi Gao Tianyu Hua Zihui Xue Yonglong Tian Shengfeng He Hang Zhao 44 53 0 23 Jun 2021
TokenLearner: What Can 8 Learned Tokens Do for Images and Videos? Michael S. Ryoo A. Piergiovanni Anurag Arnab Mostafa Dehghani A. Angelova ViT 29 127 0 21 Jun 2021
KVT: k-NN Attention for Boosting Vision Transformers Pichao Wang Xue Wang F. Wang Ming Lin Shuning Chang Hao Li R. L. Jin ViT 34 105 0 28 May 2021
Attention-based Stylisation for Exemplar Image Colourisation Marc Górriz Blanch Issa Khalifeh A. Smeaton Noel E. O'Connor M. Mrak 23 4 0 04 May 2021
MLP-Mixer: An all-MLP Architecture for Vision Ilya O. Tolstikhin N. Houlsby Alexander Kolesnikov Lucas Beyer Xiaohua Zhai ... Andreas Steiner Daniel Keysers Jakob Uszkoreit Mario Lucic Alexey Dosovitskiy 271 2,603 0 04 May 2021
On the Robustness of Vision Transformers to Adversarial Examples Kaleel Mahmood Rigel Mahmood Marten van Dijk ViT 20 217 0 31 Mar 2021
Scaling Local Self-Attention for Parameter Efficient Visual Backbones Ashish Vaswani Prajit Ramachandran A. Srinivas Niki Parmar Blake A. Hechtman Jonathon Shlens 16 395 0 23 Mar 2021
ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases Stéphane dÁscoli Hugo Touvron Matthew L. Leavitt Ari S. Morcos Giulio Biroli Levent Sagun ViT 34 803 0 19 Mar 2021
Scalable Vision Transformers with Hierarchical Pooling Zizheng Pan Bohan Zhuang Jing Liu Haoyu He Jianfei Cai ViT 25 126 0 19 Mar 2021
Involution: Inverting the Inherence of Convolution for Visual Recognition Duo Li Jie Hu Changhu Wang Xiangtai Li Qi She Lei Zhu Tong Zhang Qifeng Chen BDL 17 304 0 10 Mar 2021
Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth Yihe Dong Jean-Baptiste Cordonnier Andreas Loukas 32 373 0 05 Mar 2021
Perceiver: General Perception with Iterative Attention Andrew Jaegle Felix Gimeno Andrew Brock Andrew Zisserman Oriol Vinyals João Carreira VLM ViT MDE 48 973 0 04 Mar 2021
Generative Adversarial Transformers Drew A. Hudson C. L. Zitnick ViT 23 179 0 01 Mar 2021
LambdaNetworks: Modeling Long-Range Interactions Without Attention Irwan Bello 269 179 0 17 Feb 2021
Is Space-Time Attention All You Need for Video Understanding? Gedas Bertasius Heng Wang Lorenzo Torresani ViT 280 1,981 0 09 Feb 2021
Transformers in Vision: A Survey Salman Khan Muzammal Naseer Munawar Hayat Syed Waqas Zamir F. Khan M. Shah ViT 227 2,428 0 04 Jan 2021
GTA: Global Temporal Attention for Video Action Understanding Bo He Xitong Yang Zuxuan Wu Hao Chen Ser-Nam Lim Abhinav Shrivastava ViT 33 27 0 15 Dec 2020
Convolutional LSTM Neural Networks for Modeling Wildland Fire Dynamics J. Burge M. Bonanni M. Ihme Lily Hu 19 19 0 11 Dec 2020
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale Alexey Dosovitskiy Lucas Beyer Alexander Kolesnikov Dirk Weissenborn Xiaohua Zhai ... Matthias Minderer G. Heigold Sylvain Gelly Jakob Uszkoreit N. Houlsby ViT 41 39,217 0 22 Oct 2020
Multi-timescale Representation Learning in LSTM Language Models Shivangi Mahto Vy A. Vo Javier S. Turek Alexander G. Huth 10 29 0 27 Sep 2020
The Depth-to-Width Interplay in Self-Attention Yoav Levine Noam Wies Or Sharir Hofit Bata Amnon Shashua 22 45 0 22 Jun 2020
Neural Anisotropy Directions Guillermo Ortiz-Jiménez Apostolos Modas Seyed-Mohsen Moosavi-Dezfooli P. Frossard 26 16 0 17 Jun 2020
Visual Transformers: Token-based Image Representation and Processing for Computer Vision Bichen Wu Chenfeng Xu Xiaoliang Dai Alvin Wan Peizhao Zhang Zhicheng Yan M. Tomizuka Joseph E. Gonzalez Kurt Keutzer Peter Vajda ViT 30 545 0 05 Jun 2020
General-Purpose User Embeddings based on Mobile App Usage Junqi Zhang Bing Bai Ye Lin Jian Liang Kun Bai Fei-Yue Wang 27 35 0 27 May 2020
End-to-End Object Detection with Transformers Nicolas Carion Francisco Massa Gabriel Synnaeve Nicolas Usunier Alexander Kirillov Sergey Zagoruyko ViT 3DV PINN 71 12,671 0 26 May 2020
Convolutional Neural Networks for Sentence Classification Yoon Kim AILaw VLM 255 13,364 0 25 Aug 2014