Training data-efficient image transformers & distillation through attention

23 December 2020

Alexandre Sablayrolles

Hervé Jégou

ViT

ArXiv PDF HTML

Papers citing "Training data-efficient image transformers & distillation through attention"

31 / 1,081 papers shown

Title
Vision Transformers for Dense Prediction René Ranftl Alexey Bochkovskiy V. Koltun ViT MDE 39 1,659 0 24 Mar 2021
Multi-view 3D Reconstruction with Transformer Dan Wang Xinrui Cui Xun Chen Zhengxia Zou Tianyang Shi Septimiu Salcudean Z. J. Wang Rabab Ward ViT 20 87 0 24 Mar 2021
BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search Changlin Li Tao Tang Guangrun Wang Jiefeng Peng Bing Wang Xiaodan Liang Xiaojun Chang ViT 40 105 0 23 Mar 2021
DeepViT: Towards Deeper Vision Transformer Daquan Zhou Bingyi Kang Xiaojie Jin Linjie Yang Xiaochen Lian Zihang Jiang Qibin Hou Jiashi Feng ViT 42 510 0 22 Mar 2021
Incorporating Convolution Designs into Visual Transformers Kun Yuan Shaopeng Guo Ziwei Liu Aojun Zhou F. Yu Wei Wu ViT 24 467 0 22 Mar 2021
ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases Stéphane dÁscoli Hugo Touvron Matthew L. Leavitt Ari S. Morcos Giulio Biroli Levent Sagun ViT 31 803 0 19 Mar 2021
Scalable Vision Transformers with Hierarchical Pooling Zizheng Pan Bohan Zhuang Jing Liu Haoyu He Jianfei Cai ViT 25 126 0 19 Mar 2021
UNETR: Transformers for 3D Medical Image Segmentation Ali Hatamizadeh Yucheng Tang Vishwesh Nath Dong Yang Andriy Myronenko Bennett Landman H. Roth Daguang Xu ViT MedIm 54 1,533 0 18 Mar 2021
3D Human Pose Estimation with Spatial and Temporal Transformers Ce Zheng Sijie Zhu Matías Mendieta Taojiannan Yang C. L. P. Chen Zhengming Ding ViT 39 437 0 18 Mar 2021
TransFG: A Transformer Architecture for Fine-grained Recognition Ju He Jieneng Chen Shuai Liu Adam Kortylewski Cheng Yang Yutong Bai Changhu Wang ViT 33 375 0 14 Mar 2021
Unknown Object Segmentation from Stereo Images M. Durner W. Boerdijk M. Sundermeyer W. Friedl Zoltán-Csaba Márton Rudolph Triebel 28 34 0 11 Mar 2021
TransMed: Transformers Advance Multi-modal Medical Image Classification Yin Dai Yifan Gao ViT MedIm 30 280 0 10 Mar 2021
Pretrained Transformers as Universal Computation Engines Kevin Lu Aditya Grover Pieter Abbeel Igor Mordatch 26 217 0 09 Mar 2021
Perceiver: General Perception with Iterative Attention Andrew Jaegle Felix Gimeno Andrew Brock Andrew Zisserman Oriol Vinyals João Carreira VLM ViT MDE 48 973 0 04 Mar 2021
Transformer in Transformer Kai Han An Xiao Enhua Wu Jianyuan Guo Chunjing Xu Yunhe Wang ViT 284 1,524 0 27 Feb 2021
On Interaction Between Augmentations and Corruptions in Natural Corruption Robustness Eric Mintun A. Kirillov Saining Xie 20 89 0 22 Feb 2021
LambdaNetworks: Modeling Long-Range Interactions Without Attention Irwan Bello 269 179 0 17 Feb 2021
AlphaNet: Improved Training of Supernets with Alpha-Divergence Dilin Wang Chengyue Gong Meng Li Qiang Liu Vikas Chandra 147 44 0 16 Feb 2021
High-Performance Large-Scale Image Recognition Without Normalization Andrew Brock Soham De Samuel L. Smith Karen Simonyan VLM 223 512 0 11 Feb 2021
Is Space-Time Attention All You Need for Video Understanding? Gedas Bertasius Heng Wang Lorenzo Torresani ViT 280 1,981 0 09 Feb 2021
Bottleneck Transformers for Visual Recognition A. Srinivas Tsung-Yi Lin Niki Parmar Jonathon Shlens Pieter Abbeel Ashish Vaswani SLR 290 979 0 27 Jan 2021
Transformers in Vision: A Survey Salman Khan Muzammal Naseer Munawar Hayat Syed Waqas Zamir F. Khan M. Shah ViT 227 2,428 0 04 Jan 2021
Concept Generalization in Visual Representation Learning Mert Bulent Sariyildiz Yannis Kalantidis Diane Larlus Alahari Karteek SSL 21 50 0 10 Dec 2020
IMAGO: A family photo album dataset for a socio-historical analysis of the twentieth century L. Stacchio Alessia Angeli G. Lisanti Daniela Calanca Gustavo Marfia 23 2 0 03 Dec 2020
Contextualizing Enhances Gradient Based Meta Learning Evan Vogelbaum Rumen Dangovski L. Jing Marin Soljacic 26 3 0 17 Jul 2020
Fixing the train-test resolution discrepancy: FixEfficientNet Hugo Touvron Andrea Vedaldi Matthijs Douze Hervé Jégou AAML 189 110 0 18 Mar 2020
A Survey on 3D Skeleton-Based Action Recognition Using Learning Method Bin Ren Mengyuan Liu Runwei Ding Hong Liu 21 121 0 14 Feb 2020
Deep Weakly-Supervised Learning Methods for Classification and Localization in Histology Images: A Survey Jérôme Rony Soufiane Belharbi Jose Dolz Ismail Ben Ayed Luke McCaffrey Eric Granger 23 70 0 08 Sep 2019
Natural Adversarial Examples Dan Hendrycks Kevin Zhao Steven Basart Jacob Steinhardt D. Song OODD 47 1,417 0 16 Jul 2019
Bag of Tricks for Image Classification with Convolutional Neural Networks Tong He Zhi-Li Zhang Hang Zhang Zhongyue Zhang Junyuan Xie Mu Li 221 1,399 0 04 Dec 2018
On a Sparse Shortcut Topology of Artificial Neural Networks Fenglei Fan Dayang Wang Hengtao Guo Qikui Zhu Pingkun Yan Ge Wang Hengyong Yu 38 21 0 22 Nov 2018