Training data-efficient image transformers & distillation through attention

23 December 2020

Alexandre Sablayrolles

Hervé Jégou

ViT

ArXiv PDF HTML

Papers citing "Training data-efficient image transformers & distillation through attention"

50 / 1,080 papers shown

Title
Global-to-Local Modeling for Video-based 3D Human Pose and Shape Estimation Xi Shen Zongxin Yang Xiaohan Wang Jianxin Ma Chang Zhou Yezhou Yang ViT 3DH 21 33 0 26 Mar 2023
Towards Accurate Post-Training Quantization for Vision Transformer Yifu Ding Haotong Qin Qing-Yu Yan Z. Chai Junjie Liu Xiaolin K. Wei Xianglong Liu MQ 54 68 0 25 Mar 2023
FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterization Pavan Kumar Anasosalu Vasu J. Gabriel Jeff J. Zhu Oncel Tuzel Anurag Ranjan ViT 31 151 0 24 Mar 2023
Enhancing Multiple Reliability Measures via Nuisance-extended Information Bottleneck Jongheon Jeong Sihyun Yu Hankook Lee Jinwoo Shin AAML 36 0 0 24 Mar 2023
Exploiting Unlabelled Photos for Stronger Fine-Grained SBIR Aneeshan Sain A. Bhunia Subhadeep Koley Pinaki Nath Chowdhury Soumitri Chattopadhyay Tao Xiang Yi-Zhe Song 20 18 0 24 Mar 2023
From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labels Zhendong Yang Ailing Zeng Zhe Li Tianke Zhang Chun Yuan Yu Li 21 72 0 23 Mar 2023
Machine Learning for Brain Disorders: Transformers and Visual Transformers Robin Courant Maika Edberg Nicolas Dufour Vicky Kalogeiton MedIm ViT 27 1 0 21 Mar 2023
Transformers in Speech Processing: A Survey S. Latif Aun Zaidi Heriberto Cuayáhuitl Fahad Shamshad Moazzam Shoukat Junaid Qadir 42 47 0 21 Mar 2023
One-to-Few Label Assignment for End-to-End Dense Detection Shuai Li Minghan Li Ruihuang Li Chenhang He Lei Zhang 25 19 0 21 Mar 2023
Spatial-Aware Token for Weakly Supervised Object Localization Ping Wu Wei Zhai Yang Cao Jiebo Luo Zhengjun Zha WSOL 34 9 0 18 Mar 2023
Towards Diverse Binary Segmentation via A Simple yet General Gated Network Xiaoqi Zhao Youwei Pang Lihe Zhang Huchuan Lu Lei Zhang 23 14 0 18 Mar 2023
Depth Super-Resolution from Explicit and Implicit High-Frequency Features Xin Qiao Chenyang Ge Youming Zhang Yanhui Zhou Fabio Tosi Matteo Poggi S. Mattoccia SupR MDE 19 6 0 16 Mar 2023
Pretrained ViTs Yield Versatile Representations For Medical Images Christos Matsoukas Johan Fredin Haslum Magnus P Soderberg Kevin Smith MedIm ViT 19 11 0 13 Mar 2023
Transformer-based Image Generation from Scene Graphs Renato Sortino S. Palazzo C. Spampinato ViT 51 15 0 08 Mar 2023
SANDFORMER: CNN and Transformer under Gated Fusion for Sand Dust Image Restoration Jun-Jie Shi Bingcai Wei Gang Zhou Liye Zhang ViT 25 3 0 08 Mar 2023
CUDA: Convolution-based Unlearnable Datasets Vinu Sankar Sadasivan Mahdi Soltanolkotabi S. Feizi MU 29 23 0 07 Mar 2023
Training-Free Acceleration of ViTs with Delayed Spatial Merging J. Heo Seyedarmin Azizi A. Fayyazi Massoud Pedram 36 3 0 04 Mar 2023
Self-attention in Vision Transformers Performs Perceptual Grouping, Not Attention Paria Mehrani John K. Tsotsos 25 24 0 02 Mar 2023
Token Contrast for Weakly-Supervised Semantic Segmentation Lixiang Ru Heliang Zheng Yibing Zhan Bo Du ViT 37 86 0 02 Mar 2023
Efficient and Explicit Modelling of Image Hierarchies for Image Restoration Yawei Li Yuchen Fan Xiaoyu Xiang D. Demandolx Rakesh Ranjan Radu Timofte Luc Van Gool 18 173 0 01 Mar 2023
Capturing the motion of every joint: 3D human pose and shape estimation with independent tokens Sen Yang Wen Heng Gang Liu Guozhong Luo Wankou Yang Gang Yu 3DH ViT 18 11 0 01 Mar 2023
Human MotionFormer: Transferring Human Motions with Vision Transformers Hongyu Liu Xintong Han Chengbin Jin Lihui Qian Huawei Wei ... Faqiang Wang Haoye Dong Yibing Song Jia Xu Qifeng Chen 11 10 0 22 Feb 2023
DISCO: Distributed Inference with Sparse Communications Minghai Qin Chaowen Sun Jaco A. Hofmann D. Vučinić FedML 25 1 0 22 Feb 2023
Device Tuning for Multi-Task Large Model Penghao Jiang Xuanchen Hou Y. Zhou 21 0 0 21 Feb 2023
LIT-Former: Linking In-plane and Through-plane Transformers for Simultaneous CT Image Denoising and Deblurring Zhihao Chen Chuang Niu Qi Gao Ge Wang Hongming Shan MedIm ViT 3DV 28 20 0 21 Feb 2023
Soft Error Reliability Analysis of Vision Transformers Xing-xiong Xue Cheng Liu Ying Wang Bing Yang Tao Luo L. Zhang Huawei Li Xiaowei Li 34 14 0 21 Feb 2023
STB-VMM: Swin Transformer Based Video Motion Magnification Ricard Lado-Roigé M. A. Pérez 16 13 0 20 Feb 2023
Evaluating Representations with Readout Model Switching Yazhe Li J. Bornschein Marcus Hutter 22 0 0 19 Feb 2023
MedViT: A Robust Vision Transformer for Generalized Medical Image Classification Omid Nejati Manzari Hamid Ahmadabadi Hossein Kashiani S. B. Shokouhi Ahmad Ayatollahi ViT MedIm 21 176 0 19 Feb 2023
Hyneter: Hybrid Network Transformer for Object Detection Dong Chen Duoqian Miao Xuepeng Zhao ViT 27 3 0 18 Feb 2023
Transformadores: Fundamentos teoricos y Aplicaciones J. D. L. Torre 70 0 0 18 Feb 2023
Efficiency 360: Efficient Vision Transformers Badri N. Patro Vijay Srinivas Agneeswaran 21 6 0 16 Feb 2023
With Shared Microexponents, A Little Shifting Goes a Long Way Bita Darvish Rouhani Ritchie Zhao V. Elango Rasoul Shafipour Mathew Hall ... Eric S. Chung Zhaoxia Deng S. Naghshineh Jongsoo Park Maxim Naumov MQ 38 36 0 16 Feb 2023
From paintbrush to pixel: A review of deep neural networks in AI-generated art Anne-Sofie Maerten Derya Soydaner 34 22 0 14 Feb 2023
Bi-directional Masks for Efficient N:M Sparse Training Yu-xin Zhang Yiting Luo Mingbao Lin Yunshan Zhong Jingjing Xie Fei Chao Rongrong Ji 44 15 0 13 Feb 2023
A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexity Hongkang Li M. Wang Sijia Liu Pin-Yu Chen ViT MLT 35 56 0 12 Feb 2023
Key Design Choices for Double-Transfer in Source-Free Unsupervised Domain Adaptation Andrea Maracani Raffaello Camoriano Elisa Maiettini Davide Talon Lorenzo Rosasco Lorenzo Natale 21 2 0 10 Feb 2023
Making Substitute Models More Bayesian Can Enhance Transferability of Adversarial Examples Qizhang Li Yiwen Guo W. Zuo Hao Chen AAML 27 35 0 10 Feb 2023
Efficient Attention via Control Variates Lin Zheng Jianbo Yuan Chong-Jun Wang Lingpeng Kong 26 18 0 09 Feb 2023
Effective Data Augmentation With Diffusion Models Brandon Trabucco Kyle Doherty Max Gurinas Ruslan Salakhutdinov VLM DiffM 25 231 0 07 Feb 2023
SimCon Loss with Multiple Views for Text Supervised Semantic Segmentation Yash J. Patel Yusheng Xie Yi Zhu Srikar Appalaraju R. Manmatha 27 4 0 07 Feb 2023
High-Resolution GAN Inversion for Degraded Images in Large Diverse Datasets Yanbo Wang Chuming Lin Donghao Luo Ying Tai Zhizhong Zhang Yuan Xie 18 6 0 07 Feb 2023
AMD-HookNet for Glacier Front Segmentation Fei Wu Nora Gourmelon T. Seehaus Jianlin Zhang M. Braun Andreas K. Maier Vincent Christlein 19 9 0 06 Feb 2023
Single Cells Are Spatial Tokens: Transformers for Spatial Transcriptomic Data Imputation Haifang Wen Wenzhuo Tang Wei Jin Jiayuan Ding Renming Liu Xinnan Dai Feng Shi Lulu Shang Jiliang Tang Yuying Xie 27 8 0 06 Feb 2023
Oscillation-free Quantization for Low-bit Vision Transformers Shi Liu Zechun Liu Kwang-Ting Cheng MQ 13 34 0 04 Feb 2023
A Survey on Efficient Training of Transformers Bohan Zhuang Jing Liu Zizheng Pan Haoyu He Yuetian Weng Chunhua Shen 20 47 0 02 Feb 2023
Test-Time Amendment with a Coarse Classifier for Fine-Grained Classification Kanishk Jain Shyamgopal Karthik Vineet Gandhi 19 5 0 01 Feb 2023
Inference Time Evidences of Adversarial Attacks for Forensic on Transformers Hugo Lemarchant Liang Li Yiming Qian Yuta Nakashima Hajime Nagahara ViT AAML 38 0 0 31 Jan 2023
Adaptive Computation with Elastic Input Sequence Fuzhao Xue Valerii Likhosherstov Anurag Arnab N. Houlsby Mostafa Dehghani Yang You 29 18 0 30 Jan 2023
Diverse, Difficult, and Odd Instances (D2O): A New Test Set for Object Classification Ali Borji VLM 37 0 0 29 Jan 2023