Training data-efficient image transformers & distillation through attention

23 December 2020

Alexandre Sablayrolles

Hervé Jégou

ViT

ArXiv PDF HTML

Papers citing "Training data-efficient image transformers & distillation through attention"

50 / 1,051 papers shown

Title
Video Frame Interpolation Transformer Zhihao Shi Xiangyu Xu Xiaohong Liu Jun Chen Ming-Hsuan Yang ViT 11 157 0 27 Nov 2021
SWAT: Spatial Structure Within and Among Tokens Kumara Kahatapitiya Michael S. Ryoo 23 6 0 26 Nov 2021
VL-LTR: Learning Class-wise Visual-Linguistic Representation for Long-Tailed Visual Recognition Changyao Tian Wenhai Wang Xizhou Zhu Jifeng Dai Yu Qiao VLM 30 69 0 26 Nov 2021
Exploiting Both Domain-specific and Invariant Knowledge via a Win-win Transformer for Unsupervised Domain Adaptation Wen-hui Ma Jinming Zhang Shuang Li Chi Harold Liu Yulin Wang Wei Li ViT 16 11 0 25 Nov 2021
Domain Prompt Learning for Efficiently Adapting CLIP to Unseen Domains X. Zhang S. Gu Yutaka Matsuo Yusuke Iwasawa VLM 30 36 0 25 Nov 2021
PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers Xiaoyi Dong Jianmin Bao Ting Zhang Dongdong Chen Weiming Zhang Lu Yuan Dong Chen Fang Wen Nenghai Yu Baining Guo ViT 33 238 0 24 Nov 2021
Self-slimmed Vision Transformer Zhuofan Zong Kunchang Li Guanglu Song Yali Wang Yu Qiao B. Leng Yu Liu ViT 16 30 0 24 Nov 2021
An Image Patch is a Wave: Phase-Aware Vision MLP Yehui Tang Kai Han Jianyuan Guo Chang Xu Yanxi Li Chao Xu Yunhe Wang 22 133 0 24 Nov 2021
PhysFormer: Facial Video-based Physiological Measurement with Temporal Difference Transformer Zitong Yu Yuming Shen Jingang Shi Hengshuang Zhao Philip H. S. Torr Guoying Zhao ViT MedIm 132 167 0 23 Nov 2021
Sparse Fusion for Multimodal Transformers Yi Ding Alex Rich Mason Wang Noah Stier M. Turk P. Sen Tobias Höllerer ViT 27 7 0 23 Nov 2021
Pruning Self-attentions into Convolutional Layers in Single Path Haoyu He Jianfei Cai Jing Liu Zizheng Pan Jing Zhang Dacheng Tao Bohan Zhuang ViT 29 40 0 23 Nov 2021
Efficient Video Transformers with Spatial-Temporal Token Selection Junke Wang Xitong Yang Hengduo Li Li Liu Zuxuan Wu Yu-Gang Jiang ViT 14 63 0 23 Nov 2021
Benchmarking Detection Transfer Learning with Vision Transformers Yanghao Li Saining Xie Xinlei Chen Piotr Dollar Kaiming He Ross B. Girshick 12 164 0 22 Nov 2021
DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion Arthur Douillard Alexandre Ramé Guillaume Couairon Matthieu Cord CLL 30 295 0 22 Nov 2021
PointMixer: MLP-Mixer for Point Cloud Understanding Jaesung Choe Chunghyun Park François Rameau Jaesik Park In So Kweon 3DPC 32 98 0 22 Nov 2021
Efficient Softmax Approximation for Deep Neural Networks with Attention Mechanism Ihor Vasyltsov Wooseok Chang 16 12 0 21 Nov 2021
Discrete Representations Strengthen Vision Transformer Robustness Chengzhi Mao Lu Jiang Mostafa Dehghani Carl Vondrick Rahul Sukthankar Irfan Essa ViT 25 43 0 20 Nov 2021
FBNetV5: Neural Architecture Search for Multiple Tasks in One Run Bichen Wu Chaojian Li Hang Zhang Xiaoliang Dai Peizhao Zhang Matthew Yu Jialiang Wang Yingyan Lin Peter Vajda ViT 27 23 0 19 Nov 2021
SimMIM: A Simple Framework for Masked Image Modeling Zhenda Xie Zheng-Wei Zhang Yue Cao Yutong Lin Jianmin Bao Zhuliang Yao Qi Dai Han Hu 37 1,309 0 18 Nov 2021
Swin Transformer V2: Scaling Up Capacity and Resolution Ze Liu Han Hu Yutong Lin Zhuliang Yao Zhenda Xie ... Yue Cao Zheng-Wei Zhang Li Dong Furu Wei B. Guo ViT 41 1,744 0 18 Nov 2021
Restormer: Efficient Transformer for High-Resolution Image Restoration Syed Waqas Zamir Aditya Arora Salman Khan Munawar Hayat F. Khan Ming-Hsuan Yang ViT 32 2,120 0 18 Nov 2021
Evaluating Transformers for Lightweight Action Recognition Raivo Koot Markus Hennerbichler Haiping Lu ViT 22 8 0 18 Nov 2021
Achieving Human Parity on Visual Question Answering Ming Yan Haiyang Xu Chenliang Li Junfeng Tian Bin Bi ... Ji Zhang Songfang Huang Fei Huang Luo Si Rong Jin 24 12 0 17 Nov 2021
TRIG: Transformer-Based Text Recognizer with Initial Embedding Guidance Yuefeng Tao Zhiwei Jia Runze Ma Shugong Xu ViT 17 6 0 16 Nov 2021
iBOT: Image BERT Pre-Training with Online Tokenizer Jinghao Zhou Chen Wei Huiyu Wang Wei Shen Cihang Xie Alan Yuille Tao Kong 19 709 0 15 Nov 2021
Full-attention based Neural Architecture Search using Context Auto-regression Yuan Zhou Haiyang Wang Shuwei Huo Boyu Wang 25 3 0 13 Nov 2021
Masked Autoencoders Are Scalable Vision Learners Kaiming He Xinlei Chen Saining Xie Yanghao Li Piotr Dollár Ross B. Girshick ViT TPM 296 7,434 0 11 Nov 2021
A Survey of Visual Transformers Yang Liu Yao Zhang Yixin Wang Feng Hou Jin Yuan Jiang Tian Yang Zhang Zhongchao Shi Jianping Fan Zhiqiang He 3DGS ViT 69 330 0 11 Nov 2021
FILIP: Fine-grained Interactive Language-Image Pre-Training Lewei Yao Runhu Huang Lu Hou Guansong Lu Minzhe Niu Hang Xu Xiaodan Liang Zhenguo Li Xin Jiang Chunjing Xu VLM CLIP 28 613 0 09 Nov 2021
Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling Renrui Zhang Rongyao Fang Wei Zhang Peng Gao Kunchang Li Jifeng Dai Yu Qiao Hongsheng Li VLM 184 385 0 06 Nov 2021
Class Token and Knowledge Distillation for Multi-head Self-Attention Speaker Verification Systems Victoria Mingote A. Miguel A. O. Giménez EDUARDO LLEIDA SOLANO 25 10 0 06 Nov 2021
Can Vision Transformers Perform Convolution? Shanda Li Xiangning Chen Di He Cho-Jui Hsieh ViT 24 19 0 02 Nov 2021
Projected GANs Converge Faster Axel Sauer Kashyap Chitta Jens Muller Andreas Geiger 36 233 0 01 Nov 2021
Blending Anti-Aliasing into Vision Transformer Shengju Qian Hao Shao Yi Zhu Mu Li Jiaya Jia 19 20 0 28 Oct 2021
MVT: Multi-view Vision Transformer for 3D Object Recognition Shuo Chen Tan Yu Ping Li ViT 32 43 0 25 Oct 2021
SOFT: Softmax-free Transformer with Linear Complexity Jiachen Lu Jinghan Yao Junge Zhang Martin Danelljan Hang Xu Weiguo Gao Chunjing Xu Thomas B. Schon Li Zhang 16 161 0 22 Oct 2021
Grafting Transformer on Automatically Designed Convolutional Neural Network for Hyperspectral Image Classification Xizhe Xue Haokui Zhang Bei Fang Zongwen Bai Ying Li ViT 11 22 0 21 Oct 2021
Vis-TOP: Visual Transformer Overlay Processor Wei Hu Dian Xu Zimeng Fan Fang Liu Yanxiang He BDL ViT 12 5 0 21 Oct 2021
No One Representation to Rule Them All: Overlapping Features of Training Methods Raphael Gontijo-Lopes Yann N. Dauphin E. D. Cubuk 18 60 0 20 Oct 2021
SSAST: Self-Supervised Audio Spectrogram Transformer Yuan Gong Cheng-I Jeff Lai Yu-An Chung James R. Glass ViT 30 268 0 19 Oct 2021
TransFusion: Cross-view Fusion with Transformer for 3D Human Pose Estimation Haoyu Ma Liangjian Chen Deying Kong Zhe Wang Xingwei Liu Hao Tang Xiangyi Yan Yusheng Xie Shi-yao Lin Xiaohui Xie ViT 19 61 0 18 Oct 2021
HRFormer: High-Resolution Transformer for Dense Prediction Yuhui Yuan Rao Fu Lang Huang Weihong Lin Chao Zhang Xilin Chen Jingdong Wang ViT 24 226 0 18 Oct 2021
ASFormer: Transformer for Action Segmentation Fangqiu Yi Hongyu Wen Tingting Jiang ViT 71 172 0 16 Oct 2021
Self-Supervised Learning by Estimating Twin Class Distributions Feng Wang Tao Kong Rufeng Zhang Huaping Liu Hang Li SSL 43 16 0 14 Oct 2021
Object-Region Video Transformers Roei Herzig Elad Ben-Avraham K. Mangalam Amir Bar Gal Chechik Anna Rohrbach Trevor Darrell Amir Globerson ViT 17 82 0 13 Oct 2021
Dynamic Inference with Neural Interpreters Nasim Rahaman Muhammad Waleed Gondal S. Joshi Peter V. Gehler Yoshua Bengio Francesco Locatello Bernhard Schölkopf 23 31 0 12 Oct 2021
Global Vision Transformer Pruning with Hessian-Aware Saliency Huanrui Yang Hongxu Yin Maying Shen Pavlo Molchanov Hai Helen Li Jan Kautz ViT 28 38 0 10 Oct 2021
CLIP-Adapter: Better Vision-Language Models with Feature Adapters Peng Gao Shijie Geng Renrui Zhang Teli Ma Rongyao Fang Yongfeng Zhang Hongsheng Li Yu Qiao VLM CLIP 60 982 0 09 Oct 2021
Adversarial Token Attacks on Vision Transformers Ameya Joshi Gauri Jagatap C. Hegde ViT 30 19 0 08 Oct 2021
Context-LGM: Leveraging Object-Context Relation for Context-Aware Object Recognition Mingzhou Liu Xinwei Sun Fandong Zhang Yizhou Yu Yizhou Wang 24 0 0 08 Oct 2021