Training data-efficient image transformers & distillation through attention

23 December 2020

Alexandre Sablayrolles

Hervé Jégou

ViT

ArXiv PDF HTML

Papers citing "Training data-efficient image transformers & distillation through attention"

50 / 1,004 papers shown

Title
Pay Attention to MLPs Hanxiao Liu Zihang Dai David R. So Quoc V. Le AI4CE 22 651 0 17 May 2021
Is the aspect ratio of cells important in deep learning? A robust comparison of deep learning methods for multi-scale cytopathology cell image classification: from convolutional neural networks to visual transformers Wanli Liu Chen Li M. Rahaman Tao Jiang Hongzan Sun ... Weiming Hu Hao Chen Changhao Sun Yudong Yao M. Grzegorzek 25 54 0 16 May 2021
Conformer: Local Features Coupling Global Representations for Visual Recognition Zhiliang Peng Wei Huang Shanzhi Gu Lingxi Xie Yaowei Wang Jianbin Jiao QiXiang Ye ViT 13 527 0 09 May 2021
ResMLP: Feedforward networks for image classification with data-efficient training Hugo Touvron Piotr Bojanowski Mathilde Caron Matthieu Cord Alaaeldin El-Nouby ... Gautier Izacard Armand Joulin Gabriel Synnaeve Jakob Verbeek Hervé Jégou VLM 16 654 0 07 May 2021
Beyond Self-attention: External Attention using Two Linear Layers for Visual Tasks Meng-Hao Guo Zheng-Ning Liu Tai-Jiang Mu Shimin Hu 20 472 0 05 May 2021
Visual Relationship Detection Using Part-and-Sum Transformers with Composite Queries Qingyi Dong Z. Tu Haofu Liao Yuting Zhang Vijay Mahadevan Stefano Soatto ViT 17 38 0 05 May 2021
MLP-Mixer: An all-MLP Architecture for Vision Ilya O. Tolstikhin N. Houlsby Alexander Kolesnikov Lucas Beyer Xiaohua Zhai ... Andreas Steiner Daniel Keysers Jakob Uszkoreit Mario Lucic Alexey Dosovitskiy 241 2,600 0 04 May 2021
ConTNet: Why not use convolution and transformer at the same time? Haotian Yan Zhe Li Weijian Li Changhu Wang Ming Wu Chuang Zhang ViT 12 76 0 27 Apr 2021
Vision Transformers with Patch Diversification Chengyue Gong Dilin Wang Meng Li Vikas Chandra Qiang Liu ViT 37 62 0 26 Apr 2021
Visformer: The Vision-friendly Transformer Zhengsu Chen Lingxi Xie Jianwei Niu Xuefeng Liu Longhui Wei Qi Tian ViT 109 209 0 26 Apr 2021
Diverse Image Inpainting with Bidirectional and Autoregressive Transformers Yingchen Yu Fangneng Zhan Rongliang Wu Jianxiong Pan Kaiwen Cui Shijian Lu Feiying Ma Xuansong Xie C. Miao ViT 20 148 0 26 Apr 2021
Visual Saliency Transformer Nian Liu Ni Zhang Kaiyuan Wan Ling Shao Junwei Han ViT 253 351 0 25 Apr 2021
Multiscale Vision Transformers Haoqi Fan Bo Xiong K. Mangalam Yanghao Li Zhicheng Yan Jitendra Malik Christoph Feichtenhofer ViT 19 1,219 0 22 Apr 2021
All Tokens Matter: Token Labeling for Training Better Vision Transformers Zihang Jiang Qibin Hou Li-xin Yuan Daquan Zhou Yujun Shi Xiaojie Jin Anran Wang Jiashi Feng ViT 12 203 0 22 Apr 2021
Differentiable Model Compression via Pseudo Quantization Noise Alexandre Défossez Yossi Adi Gabriel Synnaeve DiffM MQ 10 46 0 20 Apr 2021
DisCo: Remedy Self-supervised Learning on Lightweight Models with Distilled Contrastive Learning Yuting Gao Jia-Xin Zhuang Xiaowei Guo Hao Cheng Xing Sun Ke Li Feiyue Huang 31 40 0 19 Apr 2021
Data-Efficient Language-Supervised Zero-Shot Learning with Self-Distillation Rui Cheng Bichen Wu Peizhao Zhang Peter Vajda Joseph E. Gonzalez CLIP VLM 18 31 0 18 Apr 2021
Escaping the Big Data Paradigm with Compact Transformers Ali Hassani Steven Walton Nikhil Shah Abulikemu Abuduweili Jiachen Li Humphrey Shi 54 462 0 12 Apr 2021
SiT: Self-supervised vIsion Transformer Sara Atito Ali Ahmed Muhammad Awais J. Kittler ViT 31 139 0 08 Apr 2021
Going deeper with Image Transformers Hugo Touvron Matthieu Cord Alexandre Sablayrolles Gabriel Synnaeve Hervé Jégou ViT 23 986 0 31 Mar 2021
Rethinking Spatial Dimensions of Vision Transformers Byeongho Heo Sangdoo Yun Dongyoon Han Sanghyuk Chun Junsuk Choe Seong Joon Oh ViT 332 564 0 30 Mar 2021
ViViT: A Video Vision Transformer Anurag Arnab Mostafa Dehghani G. Heigold Chen Sun Mario Lucic Cordelia Schmid ViT 30 2,085 0 29 Mar 2021
On the Adversarial Robustness of Vision Transformers Rulin Shao Zhouxing Shi Jinfeng Yi Pin-Yu Chen Cho-Jui Hsieh ViT 25 137 0 29 Mar 2021
Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding Pengchuan Zhang Xiyang Dai Jianwei Yang Bin Xiao Lu Yuan Lei Zhang Jianfeng Gao ViT 23 328 0 29 Mar 2021
Vision Transformers for Dense Prediction René Ranftl Alexey Bochkovskiy V. Koltun ViT MDE 21 1,659 0 24 Mar 2021
Multi-view 3D Reconstruction with Transformer Dan Wang Xinrui Cui Xun Chen Zhengxia Zou Tianyang Shi Septimiu Salcudean Z. J. Wang Rabab Ward ViT 20 86 0 24 Mar 2021
BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search Changlin Li Tao Tang Guangrun Wang Jiefeng Peng Bing Wang Xiaodan Liang Xiaojun Chang ViT 35 105 0 23 Mar 2021
DeepViT: Towards Deeper Vision Transformer Daquan Zhou Bingyi Kang Xiaojie Jin Linjie Yang Xiaochen Lian Zihang Jiang Qibin Hou Jiashi Feng ViT 39 510 0 22 Mar 2021
Incorporating Convolution Designs into Visual Transformers Kun Yuan Shaopeng Guo Ziwei Liu Aojun Zhou F. Yu Wei Wu ViT 24 467 0 22 Mar 2021
ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases Stéphane dÁscoli Hugo Touvron Matthew L. Leavitt Ari S. Morcos Giulio Biroli Levent Sagun ViT 29 803 0 19 Mar 2021
Scalable Vision Transformers with Hierarchical Pooling Zizheng Pan Bohan Zhuang Jing Liu Haoyu He Jianfei Cai ViT 25 126 0 19 Mar 2021
UNETR: Transformers for 3D Medical Image Segmentation Ali Hatamizadeh Yucheng Tang Vishwesh Nath Dong Yang Andriy Myronenko Bennett Landman H. Roth Daguang Xu ViT MedIm 19 1,530 0 18 Mar 2021
3D Human Pose Estimation with Spatial and Temporal Transformers Ce Zheng Sijie Zhu Matías Mendieta Taojiannan Yang C. L. P. Chen Zhengming Ding ViT 39 437 0 18 Mar 2021
TransFG: A Transformer Architecture for Fine-grained Recognition Ju He Jieneng Chen Shuai Liu Adam Kortylewski Cheng Yang Yutong Bai Changhu Wang ViT 33 375 0 14 Mar 2021
Unknown Object Segmentation from Stereo Images M. Durner W. Boerdijk M. Sundermeyer W. Friedl Zoltán-Csaba Márton Rudolph Triebel 26 34 0 11 Mar 2021
TransMed: Transformers Advance Multi-modal Medical Image Classification Yin Dai Yifan Gao ViT MedIm 18 279 0 10 Mar 2021
Pretrained Transformers as Universal Computation Engines Kevin Lu Aditya Grover Pieter Abbeel Igor Mordatch 26 217 0 09 Mar 2021
Perceiver: General Perception with Iterative Attention Andrew Jaegle Felix Gimeno Andrew Brock Andrew Zisserman Oriol Vinyals João Carreira VLM ViT MDE 48 973 0 04 Mar 2021
Transformer in Transformer Kai Han An Xiao Enhua Wu Jianyuan Guo Chunjing Xu Yunhe Wang ViT 282 1,523 0 27 Feb 2021
On Interaction Between Augmentations and Corruptions in Natural Corruption Robustness Eric Mintun A. Kirillov Saining Xie 20 88 0 22 Feb 2021
LambdaNetworks: Modeling Long-Range Interactions Without Attention Irwan Bello 260 179 0 17 Feb 2021
AlphaNet: Improved Training of Supernets with Alpha-Divergence Dilin Wang Chengyue Gong Meng Li Qiang Liu Vikas Chandra 147 44 0 16 Feb 2021
High-Performance Large-Scale Image Recognition Without Normalization Andrew Brock Soham De Samuel L. Smith Karen Simonyan VLM 223 512 0 11 Feb 2021
Is Space-Time Attention All You Need for Video Understanding? Gedas Bertasius Heng Wang Lorenzo Torresani ViT 280 1,981 0 09 Feb 2021
Bottleneck Transformers for Visual Recognition A. Srinivas Tsung-Yi Lin Niki Parmar Jonathon Shlens Pieter Abbeel Ashish Vaswani SLR 270 979 0 27 Jan 2021
Concept Generalization in Visual Representation Learning Mert Bulent Sariyildiz Yannis Kalantidis Diane Larlus Alahari Karteek SSL 21 50 0 10 Dec 2020
IMAGO: A family photo album dataset for a socio-historical analysis of the twentieth century L. Stacchio Alessia Angeli G. Lisanti Daniela Calanca Gustavo Marfia 20 2 0 03 Dec 2020
Contextualizing Enhances Gradient Based Meta Learning Evan Vogelbaum Rumen Dangovski L. Jing Marin Soljacic 21 3 0 17 Jul 2020
Fixing the train-test resolution discrepancy: FixEfficientNet Hugo Touvron Andrea Vedaldi Matthijs Douze Hervé Jégou AAML 181 110 0 18 Mar 2020
A Survey on 3D Skeleton-Based Action Recognition Using Learning Method Bin Ren Mengyuan Liu Runwei Ding Hong Liu 19 121 0 14 Feb 2020