Training data-efficient image transformers & distillation through attention

23 December 2020

Alexandre Sablayrolles

Hervé Jégou

ViT

ArXiv PDF HTML

Papers citing "Training data-efficient image transformers & distillation through attention"

50 / 1,080 papers shown

Title
Contextual Learning in Fourier Complex Field for VHR Remote Sensing Images Yan Zhang Xiyuan Gao Qingyan Duan Jiaxu Leng Xiao Pu Xinbo Gao ViT 16 1 0 28 Oct 2022
Grafting Vision Transformers Jong Sung Park Kumara Kahatapitiya Donghyun Kim Shivchander Sudalairaj Quanfu Fan Michael S. Ryoo ViT 26 2 0 28 Oct 2022
AltUB: Alternating Training Method to Update Base Distribution of Normalizing Flow for Anomaly Detection Yeongmin Kim Huiwon Jang Dongkeon Lee Ho-Jin Choi 27 9 0 26 Oct 2022
M $^3$ ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-design Hanxue Liang Zhiwen Fan Rishov Sarkar Ziyu Jiang Tianlong Chen Kai Zou Yu Cheng Cong Hao Zhangyang Wang MoE 29 81 0 26 Oct 2022
SemFormer: Semantic Guided Activation Transformer for Weakly Supervised Semantic Segmentation Junliang Chen Xiaodong Zhao Cheng Luo Linlin Shen ViT 19 3 0 26 Oct 2022
Automatic Diagnosis of Myocarditis Disease in Cardiac MRI Modality using Deep Transformers and Explainable Artificial Intelligence M. Jafari A. Shoeibi Navid Ghassemi Jónathan Heras Saiguang Ling ... Shuihua Wang R. Alizadehsani Juan M Gorriz U. Acharya Hamid Alinejad-Rokny MedIm 20 11 0 26 Oct 2022
Adversarially Robust Medical Classification via Attentive Convolutional Neural Networks I. Wasserman OOD MedIm AAML 27 0 0 26 Oct 2022
Explicitly Increasing Input Information Density for Vision Transformers on Small Datasets Xiangyu Chen Ying Qin Wenju Xu A. Bur Cuncong Zhong Guanghui Wang ViT 38 3 0 25 Oct 2022
On Fine-Tuned Deep Features for Unsupervised Domain Adaptation Qian Wang T. Breckon 19 3 0 25 Oct 2022
GliTr: Glimpse Transformers with Spatiotemporal Consistency for Online Action Prediction Samrudhdhi B. Rangrej Kevin J Liang Tal Hassner James J. Clark 27 3 0 24 Oct 2022
MetaFormer Baselines for Vision Weihao Yu Chenyang Si Pan Zhou Mi Luo Yichen Zhou Jiashi Feng Shuicheng Yan Xinchao Wang MoE 23 156 0 24 Oct 2022
Delving into Masked Autoencoders for Multi-Label Thorax Disease Classification Junfei Xiao Yutong Bai Alan Yuille Zongwei Zhou MedIm ViT 32 58 0 23 Oct 2022
Adversarial Pretraining of Self-Supervised Deep Networks: Past, Present and Future Guo-Jun Qi M. Shah SSL 23 8 0 23 Oct 2022
S2WAT: Image Style Transfer via Hierarchical Vision Transformer using Strips Window Attention Chi Zhang Xiaogang Xu Lei Wang Zaiyan Dai Jun Yang ViT 27 23 0 22 Oct 2022
Face Pyramid Vision Transformer Khawar Islam M. Zaheer Arif Mahmood ViT CVBM 24 4 0 21 Oct 2022
LiteVL: Efficient Video-Language Learning with Enhanced Spatial-Temporal Modeling Dongsheng Chen Chaofan Tao Lu Hou Lifeng Shang Xin Jiang Qun Liu VLM 29 18 0 21 Oct 2022
Boosting vision transformers for image retrieval Chull Hwan Song Jooyoung Yoon Shunghyun Choi Yannis Avrithis ViT 26 31 0 21 Oct 2022
i-MAE: Are Latent Representations in Masked Autoencoders Linearly Separable? Kevin Zhang Zhiqiang Shen 20 8 0 20 Oct 2022
Play It Back: Iterative Attention for Audio Recognition Alexandros Stergiou Dima Damen 28 4 0 20 Oct 2022
Single Image Super-Resolution Using Lightweight Networks Based on Swin Transformer Bolong Zhang Juan Chen Q. Wen ViT 33 1 0 20 Oct 2022
SSiT: Saliency-guided Self-supervised Image Transformer for Diabetic Retinopathy Grading Yijin Huang Junyan Lyu Pujin Cheng Roger Tam Xiaoying Tang ViT MedIm 19 19 0 20 Oct 2022
A Unified View of Masked Image Modeling Zhiliang Peng Li Dong Hangbo Bao QiXiang Ye Furu Wei VLM 52 35 0 19 Oct 2022
ViTCoD: Vision Transformer Acceleration via Dedicated Algorithm and Accelerator Co-Design Haoran You Zhanyi Sun Huihong Shi Zhongzhi Yu Yang Katie Zhao Yongan Zhang Chaojian Li Baopu Li Yingyan Lin ViT 17 76 0 18 Oct 2022
Token Merging: Your ViT But Faster Daniel Bolya Cheng-Yang Fu Xiaoliang Dai Peizhao Zhang Christoph Feichtenhofer Judy Hoffman MoMe 28 417 0 17 Oct 2022
Contrastive Language-Image Pre-Training with Knowledge Graphs Xuran Pan Tianzhu Ye Dongchen Han S. Song Gao Huang VLM CLIP 22 43 0 17 Oct 2022
Learning Self-Regularized Adversarial Views for Self-Supervised Vision Transformers Tao Tang Changlin Li Guangrun Wang Kaicheng Yu Xiaojun Chang Xiaodan Liang ViT 18 1 0 16 Oct 2022
MoRSE: Deep Learning-based Arm Gesture Recognition for Search and Rescue Operations P. Kasnesis Christos Chatzigeorgiou Dimitrios G. Kogias C. Patrikakis Harris V. Georgiou Aspasia Tzeletopoulou 13 3 0 15 Oct 2022
Neural Attentive Circuits Nasim Rahaman M. Weiß Francesco Locatello C. Pal Yoshua Bengio Bernhard Schölkopf Erran L. Li Nicolas Ballas 24 6 0 14 Oct 2022
Sequential Learning Of Neural Networks for Prequential MDL J. Bornschein Yazhe Li Marcus Hutter AI4TS 25 6 0 14 Oct 2022
Is synthetic data from generative models ready for image recognition? Ruifei He Shuyang Sun Xin Yu Chuhui Xue Wenqing Zhang Philip H. S. Torr Song Bai Xiaojuan Qi 16 285 0 14 Oct 2022
When Adversarial Training Meets Vision Transformers: Recipes from Training to Architecture Yi Mo Dongxian Wu Yifei Wang Yiwen Guo Yisen Wang ViT 37 52 0 14 Oct 2022
S4ND: Modeling Images and Videos as Multidimensional Signals Using State Spaces Eric N. D. Nguyen Karan Goel Albert Gu Gordon W. Downs Preey Shah Tri Dao S. Baccus Christopher Ré VLM 22 38 0 12 Oct 2022
Bridging the Gap Between Vision Transformers and Convolutional Neural Networks on Small Datasets Zhiying Lu Hongtao Xie Chuanbin Liu Yongdong Zhang ViT 12 57 0 12 Oct 2022
SaiT: Sparse Vision Transformers through Adaptive Token Pruning Ling Li D. Thorsley Joseph Hassoun ViT 25 17 0 11 Oct 2022
Curved Representation Space of Vision Transformers Juyeop Kim Junha Park Songkuk Kim Jongseok Lee ViT 33 6 0 11 Oct 2022
ViewFool: Evaluating the Robustness of Visual Recognition to Adversarial Viewpoints Yinpeng Dong Shouwei Ruan Hang Su Cai Kang Xingxing Wei Junyi Zhu AAML 28 50 0 08 Oct 2022
Class-wise and reduced calibration methods Michael Panchenko Anes Benmerzoug Miguel de Benito Delgado 16 0 0 07 Oct 2022
Pose Guided Human Image Synthesis with Partially Decoupled GAN Jianguo Wu Jianzong Wang Shijing Si Xiaoyang Qu Jing Xiao 3DH 30 3 0 07 Oct 2022
Game-Theoretic Understanding of Misclassification Kosuke Sumiyasu K. Kawamoto Hiroshi Kera 26 1 0 07 Oct 2022
EmbryosFormer: Deformable Transformer and Collaborative Encoding-Decoding for Embryos Stage Development Classification Tien-Phat Nguyen Trong-Thang Pham Tri Minh Nguyen H. Le Dung Nguyen Hau Lam Phong H. Nguyen Jennifer Fowler Minh-Triet Tran Ngan Le ViT 30 13 0 07 Oct 2022
The Lie Derivative for Measuring Learned Equivariance Nate Gruver Marc Finzi Micah Goldblum A. Wilson 16 34 0 06 Oct 2022
Effective Self-supervised Pre-training on Low-compute Networks without Distillation Fuwen Tan F. Saleh Brais Martínez 27 4 0 06 Oct 2022
Active Image Indexing Pierre Fernandez Matthijs Douze Hervé Jégou Teddy Furon VLM 23 8 0 05 Oct 2022
Centralized Feature Pyramid for Object Detection Yu Quan Dong Zhang Liyan Zhang Jinhui Tang ObjD 26 147 0 05 Oct 2022
Natural Color Fool: Towards Boosting Black-box Unrestricted Attacks Shengming Yuan Qilong Zhang Lianli Gao Yaya Cheng Jingkuan Song AAML 22 42 0 05 Oct 2022
MOAT: Alternating Mobile Convolution and Attention Brings Strong Vision Models Chenglin Yang Siyuan Qiao Qihang Yu Xiaoding Yuan Yukun Zhu Alan Yuille Hartwig Adam Liang-Chieh Chen ViT MoE 30 58 0 04 Oct 2022
Implicit Warping for Animation with Image Sets Arun Mallya Ting-Chun Wang Ming-Yu Liu VGen 114 41 0 04 Oct 2022
Bridged Transformer for Vision and Point Cloud 3D Object Detection Yikai Wang Tengqi Ye Lele Cao Wen-bing Huang Fuchun Sun Fengxiang He Dacheng Tao ViT 35 34 0 04 Oct 2022
Introducing Vision Transformer for Alzheimer's Disease classification task with 3D input Zilun Zhang Farzad Khalvati MedIm ViT 20 9 0 03 Oct 2022
Early or Late Fusion Matters: Efficient RGB-D Fusion in Vision Transformers for 3D Object Recognition Georgios Tziafas H. Kasaei ViT 35 10 0 03 Oct 2022