Training data-efficient image transformers & distillation through attention

23 December 2020

Alexandre Sablayrolles

Hervé Jégou

ViT

ArXiv PDF HTML

Papers citing "Training data-efficient image transformers & distillation through attention"

50 / 1,080 papers shown

Title
CancerUniT: Towards a Single Unified Model for Effective Detection, Segmentation, and Diagnosis of Eight Major Cancers Using a Large Collection of CT Scans Jieneng Chen Yingda Xia Jiawen Yao K. Yan Jianpeng Zhang ... Xin Chen Jingren Zhou Alan Yuille Zai-De Liu Ling Zhang ViT MedIm 28 15 0 28 Jan 2023
A Closer Look at Few-shot Classification Again Xu Luo Hao Wu Ji Zhang Lianli Gao Jing Xu Jingkuan Song 24 48 0 28 Jan 2023
Cross-Architectural Positive Pairs improve the effectiveness of Self-Supervised Learning P. Singh Jacopo Cirrone SSL 40 0 0 27 Jan 2023
Enhancing Face Recognition with Latent Space Data Augmentation and Facial Posture Reconstruction Soroush Hashemifar Abdolreza Marefat Javad Hassannataj Joloudari H. Hassanpour CVBM 21 11 0 27 Jan 2023
Open Problems in Applied Deep Learning M. Raissi AI4CE 34 2 0 26 Jan 2023
Out of Distribution Performance of State of Art Vision Model Salman Rahman W. Lee 24 2 0 25 Jan 2023
StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis Axel Sauer Tero Karras S. Laine Andreas Geiger Timo Aila 24 206 0 23 Jan 2023
Curriculum Script Distillation for Multilingual Visual Question Answering Khyathi Raghavi Chandu A. Geramifard 25 0 0 17 Jan 2023
Vision Learners Meet Web Image-Text Pairs Bingchen Zhao Quan Cui Hao Wu Osamu Yoshie Cheng Yang Oisin Mac Aodha VLM 21 5 0 17 Jan 2023
Opti-CAM: Optimizing saliency maps for interpretability Hanwei Zhang Felipe Torres R. Sicre Yannis Avrithis Stéphane Ayache 28 22 0 17 Jan 2023
GOHSP: A Unified Framework of Graph and Optimization-based Heterogeneous Structured Pruning for Vision Transformer Miao Yin Burak Uzkent Yilin Shen Hongxia Jin Bo Yuan ViT 24 13 0 13 Jan 2023
Toward Building General Foundation Models for Language, Vision, and Vision-Language Understanding Tasks Xinsong Zhang Yan Zeng Jipeng Zhang Hang Li VLM AI4CE LRM 14 17 0 12 Jan 2023
Dynamic Grained Encoder for Vision Transformers Lin Song Songyang Zhang Songtao Liu Zeming Li Xuming He Hongbin Sun Jian-jun Sun Nanning Zheng ViT 26 34 0 10 Jan 2023
Does compressing activations help model parallel training? S. Bian Dacheng Li Hongyi Wang Eric P. Xing Shivaram Venkataraman 19 4 0 06 Jan 2023
Exploring Efficient Few-shot Adaptation for Vision Transformers C. Xu Siqian Yang Yabiao Wang Zhanxiong Wang Yanwei Fu Xiangyang Xue 27 16 0 06 Jan 2023
Skip-Attention: Improving Vision Transformers by Paying Less Attention Shashanka Venkataramanan Amir Ghodrati Yuki M. Asano Fatih Porikli A. Habibian ViT 15 25 0 05 Jan 2023
SPRING: Situated Conversation Agent Pretrained with Multimodal Questions from Incremental Layout Graph Yuxing Long Binyuan Hui Fulong Ye Yanyang Li Zhuoxin Han Caixia Yuan Yongbin Li Xiaojie Wang LLMAG 25 7 0 05 Jan 2023
Semi-MAE: Masked Autoencoders for Semi-supervised Vision Transformers Haojie Yu Kangnian Zhao Xiaoming Xu ViT 28 1 0 04 Jan 2023
Explainability and Robustness of Deep Visual Classification Models Jindong Gu AAML 31 2 0 03 Jan 2023
TinyMIM: An Empirical Study of Distilling MIM Pre-trained Models Sucheng Ren Fangyun Wei Zheng-Wei Zhang Han Hu 35 34 0 03 Jan 2023
Reference Twice: A Simple and Unified Baseline for Few-Shot Instance Segmentation Yue Han Jiangning Zhang Zhucun Xue Chao Xu Xintian Shen Yabiao Wang Chengjie Wang Yong Liu Xiangtai Li 29 17 0 03 Jan 2023
PanopticPartFormer++: A Unified and Decoupled View for Panoptic Part Segmentation Xiangtai Li Shilin Xu Yibo Yang Haobo Yuan Guangliang Cheng Yu Tong Zhouchen Lin Ming-Hsuan Yang Dacheng Tao ViT 37 21 0 03 Jan 2023
Edge Enhanced Image Style Transfer via Transformers Chi Zhang Jun Yang Zaiyan Dai Peng-Xia Cao 11 10 0 02 Jan 2023
Pseudo-Inverted Bottleneck Convolution for DARTS Search Space Arash Ahmadian Louis S.P. Liu Yue Fei Konstantinos N. Plataniotis Mahdi S. Hosseini 19 0 0 31 Dec 2022
Exploring Vision Transformers as Diffusion Learners He Cao Jianan Wang Tianhe Ren Xianbiao Qi Yihao Chen Yuan Yao L. Zhang 31 10 0 28 Dec 2022
OVO: One-shot Vision Transformer Search with Online distillation Zimian Wei H. Pan Xin-Yi Niu Dongsheng Li ViT 29 1 0 28 Dec 2022
Representation Separation for Semantic Segmentation with Vision Transformers Yuanduo Hong Huihui Pan Weichao Sun Xinghu Yu Huijun Gao ViT 21 5 0 28 Dec 2022
SMMix: Self-Motivated Image Mixing for Vision Transformers Mengzhao Chen Mingbao Lin Zhihang Lin Yu-xin Zhang Fei Chao Rongrong Ji 45 10 0 26 Dec 2022
A Close Look at Spatial Modeling: From Attention to Convolution Xu Ma Huan Wang Can Qin Kunpeng Li Xing Zhao Jie Fu Yun Fu ViT 3DPC 17 11 0 23 Dec 2022
Reversible Column Networks Yuxuan Cai Yi Zhou Qi Han Jianjian Sun Xiangwen Kong Jun Yu Li Xiangyu Zhang VLM 31 53 0 22 Dec 2022
Metadata-guided Consistency Learning for High Content Images Johan Fredin Haslum Christos Matsoukas Karl‐Johan Leuchowius Erik Mullers Kevin Smith 25 8 0 22 Dec 2022
Bort: Towards Explainable Neural Networks with Bounded Orthogonal Constraint Borui Zhang Wenzhao Zheng Jie Zhou Jiwen Lu AAML 23 7 0 18 Dec 2022
RepQ-ViT: Scale Reparameterization for Post-Training Quantization of Vision Transformers Zhikai Li Junrui Xiao Lianwei Yang Qingyi Gu MQ 26 81 0 16 Dec 2022
Rethinking Vision Transformers for MobileNet Size and Speed Yanyu Li Ju Hu Yang Wen Georgios Evangelidis Kamyar Salahi Yanzhi Wang Sergey Tulyakov Jian Ren ViT 25 159 0 15 Dec 2022
PD-Quant: Post-Training Quantization based on Prediction Difference Metric Jiawei Liu Lin Niu Zhihang Yuan Dawei Yang Xinggang Wang Wenyu Liu MQ 96 68 0 14 Dec 2022
Look Before You Match: Instance Understanding Matters in Video Object Segmentation Junke Wang Dongdong Chen Zuxuan Wu Chong Luo Chuanxin Tang Xiyang Dai Yucheng Zhao Yujia Xie Lu Yuan Yu-Gang Jiang VOS 28 39 0 13 Dec 2022
GPViT: A High Resolution Non-Hierarchical Vision Transformer with Group Propagation Chenhongyi Yang Jiarui Xu Shalini De Mello Elliot J. Crowley X. Wang ViT 30 21 0 13 Dec 2022
A Neural ODE Interpretation of Transformer Layers Yaofeng Desmond Zhong Tongtao Zhang Amit Chakraborty Biswadip Dey 20 9 0 12 Dec 2022
Cross-Modal Learning with 3D Deformable Attention for Action Recognition Sangwon Kim Dasom Ahn ByoungChul Ko ViT 3DPC 25 24 0 12 Dec 2022
Vision Transformer with Attentive Pooling for Robust Facial Expression Recognition Fanglei Xue Qiangchang Wang Zichang Tan Zhongsong Ma G. Guo ViT 33 67 0 11 Dec 2022
Position Embedding Needs an Independent Layer Normalization Runyi Yu Zhennan Wang Yinhuai Wang Kehan Li Yian Zhao Jian Andrew Zhang Guoli Song Jie Chen 26 1 0 10 Dec 2022
Co-training $2^L$ Submodels for Visual Recognition Hugo Touvron Matthieu Cord Maxime Oquab Piotr Bojanowski Jakob Verbeek Hervé Jégou VLM 27 9 0 09 Dec 2022
Dynamic Test-Time Augmentation via Differentiable Functions Shohei Enomoto Monikka Roslianna Busto Takeharu Eda OOD 35 5 0 09 Dec 2022
A New Linear Scaling Rule for Private Adaptive Hyperparameter Optimization Ashwinee Panda Xinyu Tang Saeed Mahloujifar Vikash Sehwag Prateek Mittal 31 11 0 08 Dec 2022
PromptonomyViT: Multi-Task Prompt Learning Improves Video Transformers using Synthetic Scene Data Roei Herzig Ofir Abramovich Elad Ben-Avraham Assaf Arbelle Leonid Karlinsky Ariel Shamir Trevor Darrell Amir Globerson 32 16 0 08 Dec 2022
Deep Incubation: Training Large Models by Divide-and-Conquering Zanlin Ni Yulin Wang Jiangwei Yu Haojun Jiang Yu Cao Gao Huang VLM 18 11 0 08 Dec 2022
Federated Learning for Inference at Anytime and Anywhere Zicheng Liu Da Li Javier Fernandez-Marques Stefanos Laskaridis Yan Gao L. Dudziak Stan Z. Li S. Hu Timothy M. Hospedales FedML 21 5 0 08 Dec 2022
Vision Transformer Computation and Resilience for Dynamic Inference Kavya Sreedhar Jason Clemons Rangharajan Venkatesan S. Keckler M. Horowitz 24 2 0 06 Dec 2022
Exploring Stochastic Autoregressive Image Modeling for Visual Representation Yu-Hang Qi Fan Yang Yousong Zhu Yufei Liu Liwei Wu Rui Zhao Wei Li DiffM 27 13 0 03 Dec 2022
Leveraging Angular Information Between Feature and Classifier for Long-tailed Learning: A Prediction Reformulation Approach Haoxu Wang Junchi Yan 19 1 0 03 Dec 2022