Training data-efficient image transformers & distillation through attention

23 December 2020

Alexandre Sablayrolles

Hervé Jégou

ViT

ArXiv PDF HTML

Papers citing "Training data-efficient image transformers & distillation through attention"

50 / 1,080 papers shown

Title
Performance-optimized deep neural networks are evolving into worse models of inferotemporal visual cortex Drew Linsley I. F. Rodriguez Thomas Fel Michael Arcaro Saloni Sharma Margaret Livingstone Thomas Serre 27 18 0 06 Jun 2023
Quantifying the Variability Collapse of Neural Networks Jing-Xue Xu Haoxiong Liu 31 4 0 06 Jun 2023
Centered Self-Attention Layers Ameen Ali Tomer Galanti Lior Wolf 28 6 0 02 Jun 2023
Lightweight Vision Transformer with Bidirectional Interaction Qihang Fan Huaibo Huang Xiaoqiang Zhou Ran He ViT 37 28 0 01 Jun 2023
CVSNet: A Computer Implementation for Central Visual System of The Brain Ruimin Gao Hao-Li Zou Zhekai Duan 26 3 0 31 May 2023
Vision Transformers for Mobile Applications: A Short Survey Nahid Alam Steven Kolawole S. Sethi Nishant Bansali Karina Nguyen ViT 18 3 0 30 May 2023
HaVQA: A Dataset for Visual Question Answering and Multimodal Research in Hausa Language Shantipriya Parida Idris Abdulmumin Shamsuddeen Hassan Muhammad Aneesh Bose Guneet Singh Kohli I. Ahmad Ketan Kotwal S. Sarkar Ondrej Bojar Habeebah Adamu Kakudi 22 4 0 28 May 2023
Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformers Hongjie Wang Bhishma Dedhia N. Jha ViT VLM 36 26 0 27 May 2023
COMCAT: Towards Efficient Compression and Customization of Attention-Based Vision Models Jinqi Xiao Miao Yin Yu Gong Xiao Zang Jian Ren Bo Yuan VLM ViT 30 9 0 26 May 2023
Neural Foundations of Mental Simulation: Future Prediction of Latent Representations on Dynamic Scenes Aran Nayebi R. Rajalingham M. Jazayeri G. R. Yang 36 17 0 19 May 2023
SlotDiffusion: Object-Centric Generative Modeling with Diffusion Models Ziyi Wu Jingyu Hu Wuyue Lu Igor Gilitschenski Animesh Garg DiffM OCL 30 44 0 18 May 2023
Student-friendly Knowledge Distillation Mengyang Yuan Bo Lang Fengnan Quan 18 17 0 18 May 2023
Adversarial Amendment is the Only Force Capable of Transforming an Enemy into a Friend Chong Yu Tao Chen Zhongxue Gan AAML 7 1 0 18 May 2023
Boost Vision Transformer with GPU-Friendly Sparsity and Quantization Chong Yu Tao Chen Zhongxue Gan Jiayuan Fan MQ ViT 25 23 0 18 May 2023
CageViT: Convolutional Activation Guided Efficient Vision Transformer Hao Zheng Jinbao Wang Xiantong Zhen H. Chen Jingkuan Song Feng Zheng ViT 10 0 0 17 May 2023
OneCAD: One Classifier for All image Datasets using multimodal learning S. Wadekar Eugenio Culurciello 32 0 0 11 May 2023
Patch-wise Mixed-Precision Quantization of Vision Transformer Junrui Xiao Zhikai Li Lianwei Yang Qingyi Gu MQ 22 12 0 11 May 2023
Transformer-Based Model for Monocular Visual Odometry: A Video Understanding Approach André O. Françani Marcos R. O. A. Máximo 25 8 0 10 May 2023
Adapt and Align to Improve Zero-Shot Sketch-Based Image Retrieval Shiyin Dong Mingrui Zhu N. Wang Xinbo Gao VLM 27 3 0 09 May 2023
MMViT: Multiscale Multiview Vision Transformers Yuchen Liu Natasha Ong Kaiyan Peng Bo Xiong Qifan Wang ... Madian Khabsa Kaiyue Yang David C. Liu Donald Williamson Hanchao Yu ViT 22 4 0 28 Apr 2023
iMixer: hierarchical Hopfield network implies an invertible, implicit and iterative MLP-Mixer Toshihiro Ota Masato Taki 27 2 0 25 Apr 2023
Self-supervised Learning by View Synthesis Shaoteng Liu Xiangyu Zhang T. Hu Jiaya Jia 3DV ViT 40 1 0 22 Apr 2023
Efficient Video Action Detection with Token Dropout and Context Refinement Lei Chen Zhan Tong Yibing Song Gangshan Wu Limin Wang 36 14 0 17 Apr 2023
Permutation Equivariance of Transformers and Its Applications Hengyuan Xu Liyao Xiang Hang Ye Dixi Yao Pengzhi Chu Baochun Li 17 13 0 16 Apr 2023
MA-ViT: Modality-Agnostic Vision Transformers for Face Anti-Spoofing Ajian Liu Yanyan Liang 44 48 0 15 Apr 2023
PARFormer: Transformer-based Multi-Task Network for Pedestrian Attribute Recognition Xinwen Fan Yukang Zhang Yang Lu Hanzi Wang ViT 19 29 0 14 Apr 2023
Preserving Locality in Vision Transformers for Class Incremental Learning Bowen Zheng Da-Wei Zhou Han-Jia Ye De-Chuan Zhan CLL 19 5 0 14 Apr 2023
Zoom-VQA: Patches, Frames and Clips Integration for Video Quality Assessment Kai Zhao Kun Yuan Ming-Ting Sun Xingsen Wen 10 20 0 13 Apr 2023
TransHP: Image Classification with Hierarchical Prompting Wenhao Wang Yifan Sun W. Li Yi Yang 19 16 0 13 Apr 2023
RIFormer: Keep Your Vision Backbone Effective While Removing Token Mixer Jiahao Wang Songyang Zhang Yong Liu Taiqiang Wu Yujiu Yang Xihui Liu Kai-xiang Chen Ping Luo Dahua Lin 24 20 0 12 Apr 2023
Distilling Token-Pruned Pose Transformer for 2D Human Pose Estimation Feixiang Ren ViT 19 2 0 12 Apr 2023
Data-Efficient Image Quality Assessment with Attention-Panel Decoder Guanyi Qin R. Hu Yutao Liu Xiawu Zheng Haotian Liu Xiu Li Yan Zhang ViT 21 60 0 11 Apr 2023
A Comprehensive Survey on Knowledge Distillation of Diffusion Models Weijian Luo DiffM MedIm 49 33 0 09 Apr 2023
Token Boosting for Robust Self-Supervised Visual Transformer Pre-training Tianjiao Li Lin Geng Foo Ping Hu Xindi Shang Hossein Rahmani Zehuan Yuan J. Liu 32 7 0 09 Apr 2023
SwiftTron: An Efficient Hardware Accelerator for Quantized Transformers Alberto Marchisio David Durà Maurizio Capra Maurizio Martina Guido Masera Muhammad Shafique 21 17 0 08 Apr 2023
On Efficient Training of Large-Scale Deep Learning Models: A Literature Review Li Shen Yan Sun Zhiyuan Yu Liang Ding Xinmei Tian Dacheng Tao VLM 28 40 0 07 Apr 2023
From Saliency to DINO: Saliency-guided Vision Transformer for Few-shot Keypoint Detection Changsheng Lu Hao Zhu Piotr Koniusz 42 11 0 06 Apr 2023
DIR-AS: Decoupling Individual Identification and Temporal Reasoning for Action Segmentation Peiyao Wang Haibin Ling 15 2 0 04 Apr 2023
Revisiting the Evaluation of Image Synthesis with GANs Mengping Yang Ceyuan Yang Yichi Zhang Qingyan Bai Yujun Shen Bo Dai EGVM 27 7 0 04 Apr 2023
Adaptive Sparse Pairwise Loss for Object Re-Identification Xiao Zhou Yujie Zhong Zhen Cheng Fan Liang Lin Ma 11 39 0 31 Mar 2023
SparseViT: Revisiting Activation Sparsity for Efficient High-Resolution Vision Transformer Xuanyao Chen Zhijian Liu Haotian Tang Li Yi Hang Zhao Song Han ViT 21 46 0 30 Mar 2023
Soft Neighbors are Positive Supporters in Contrastive Visual Representation Learning Chongjian Ge Jiangliu Wang Zhan Tong Shoufa Chen Yibing Song Ping Luo SSL 22 27 0 30 Mar 2023
Multi-scale Hierarchical Vision Transformer with Cascaded Attention Decoding for Medical Image Segmentation Md Mostafijur Rahman R. Marculescu MedIm ViT 19 43 0 29 Mar 2023
InceptionNeXt: When Inception Meets ConvNeXt Weihao Yu Pan Zhou Shuicheng Yan Xinchao Wang 45 117 0 29 Mar 2023
SELF-VS: Self-supervised Encoding Learning For Video Summarization Hojjat Mokhtarabadi Kaveh Bahraman M. Hosseinzadeh M. Eftekhari AI4TS SSL ViT 25 0 0 28 Mar 2023
Transferable Adversarial Attacks on Vision Transformers with Token Gradient Regularization Jianping Zhang Yizhan Huang Weibin Wu Michael R. Lyu AAML ViT 18 49 0 28 Mar 2023
D-TrAttUnet: Dual-Decoder Transformer-Based Attention Unet Architecture for Binary and Multi-classes Covid-19 Infection Segmentation F. Bougourzi C. Distante Fadi Dornaika A. Taleb-Ahmed MedIm ViT 9 5 0 27 Mar 2023
SwiftFormer: Efficient Additive Attention for Transformer-based Real-time Mobile Vision Applications Abdelrahman M. Shaker Muhammad Maaz H. Rasheed Salman Khan Ming Yang F. Khan ViT 40 84 0 27 Mar 2023
Vision Transformer with Quadrangle Attention Qiming Zhang Jing Zhang Yufei Xu Dacheng Tao ViT 19 38 0 27 Mar 2023
Learned Image Compression with Mixed Transformer-CNN Architectures Jinming Liu Heming Sun J. Katto 10 220 0 27 Mar 2023