Training data-efficient image transformers & distillation through attention

23 December 2020

Alexandre Sablayrolles

Hervé Jégou

ViT

ArXiv PDF HTML

Papers citing "Training data-efficient image transformers & distillation through attention"

50 / 983 papers shown

Title
Generalization in birdsong classification: impact of transfer learning methods and dataset characteristics Burooj Ghani Vincent J. Kalkman Bob Planqué Willem-Pier Vellinga L. Gill Dan Stowell VLM 24 5 0 21 Sep 2024
Multi-OCT-SelfNet: Integrating Self-Supervised Learning with Multi-Source Data Fusion for Enhanced Multi-Class Retinal Disease Classification Fatema Jannat Sina Gholami Jennifer I. Lim Theodore Leng Minhaj Nur Alam Hamed Tabkhi 28 0 0 17 Sep 2024
SkinMamba: A Precision Skin Lesion Segmentation Architecture with Cross-Scale Global State Modeling and Frequency Boundary Guidance Shun Zou Mingya Zhang Bingjian Fan Zhengyi Zhou Xiuguo Zou Mamba 24 3 0 17 Sep 2024
Frequency-Guided Masking for Enhanced Vision Self-Supervised Learning Amin Karimi Monsefi Mengxi Zhou Nastaran Karimi Monsefi Ser-Nam Lim Wei-Lun Chao R. Ramnath 36 1 0 16 Sep 2024
Effective Pre-Training of Audio Transformers for Sound Event Detection Florian Schmid T. Morocutti Francesco Foscarin Jan Schluter Paul Primus Gerhard Widmer ViT 23 2 0 14 Sep 2024
Token Turing Machines are Efficient Vision Models Purvish Jajal Nick Eliopoulos Benjamin Shiue-Hal Chou George K. Thiravathukal James C. Davis Yung-Hsiang Lu 85 0 0 11 Sep 2024
Brain-Inspired Stepwise Patch Merging for Vision Transformers Yonghao Yu Dongcheng Zhao Guobin Shen Yiting Dong Yi Zeng 45 0 0 11 Sep 2024
DetailCLIP: Detail-Oriented CLIP for Fine-Grained Tasks Amin Karimi Monsefi Kishore Prakash Sailaja Ali Alilooee Ser-Nam Lim R. Ramnath VLM 33 6 0 10 Sep 2024
Cross-attention Inspired Selective State Space Models for Target Sound Extraction Donghang Wu Yiwen Wang Xihong Wu T. Qu Mamba 26 3 0 07 Sep 2024
Learning to Discover Forgery Cues for Face Forgery Detection Jiahe Tian Peng-Wen Chen Cai Yu Xiaomeng Fu Xi Wang Jiao Dai Jizhong Han CVBM AAML 29 6 0 02 Sep 2024
A Survey of the Self Supervised Learning Mechanisms for Vision Transformers Asifullah Khan A. Sohail M. Fiaz Mehdi Hassan Tariq Habib Afridi ... Muhammad Zaigham Zaheer Kamran Ali Tangina Sultana Ziaurrehman Tanoli Naeem Akhter 43 3 0 30 Aug 2024
MePT: Multi-Representation Guided Prompt Tuning for Vision-Language Model Xinyang Wang Yi Yang Minfeng Zhu Kecheng Zheng Shi Liu Wei Chen VPVLM MLLM VLM 41 1 0 19 Aug 2024
FEDKIM: Adaptive Federated Knowledge Injection into Medical Foundation Models Xiaochen Wang Jiaqi Wang Houping Xiao J. Chen Fenglong Ma MedIm 61 7 0 17 Aug 2024
Breaking Class Barriers: Efficient Dataset Distillation via Inter-Class Feature Compensator Xin Zhang Jiawei Du Ping Liu Joey Tianyi Zhou DD 42 2 0 13 Aug 2024
Attacks and Defenses for Generative Diffusion Models: A Comprehensive Survey V. T. Truong Luan Ba Dang Long Bao Le DiffM MedIm 42 16 0 06 Aug 2024
Modelling Visual Semantics via Image Captioning to extract Enhanced Multi-Level Cross-Modal Semantic Incongruity Representation with Attention for Multimodal Sarcasm Detection Sajal Aggarwal Ananya Pandey Dinesh Kumar Vishwakarma 41 1 0 05 Aug 2024
Unsupervised Representation Learning by Balanced Self Attention Matching Daniel Shalam Simon Korman SSL 33 0 0 04 Aug 2024
DeMansia: Mamba Never Forgets Any Tokens Ricky Fang Mamba 19 0 0 04 Aug 2024
Scaling Backwards: Minimal Synthetic Pre-training? Ryo Nakamura Ryu Tadokoro Ryosuke Yamada Tim Puhlfürß Iro Laina Christian Rupprecht Walid Maalej Rio Yokota Hirokatsu Kataoka DD 16 2 0 01 Aug 2024
MimiQ: Low-Bit Data-Free Quantization of Vision Transformers with Encouraging Inter-Head Attention Similarity Kanghyun Choi Hyeyoon Lee Dain Kwon Sunjong Park Kyuyeun Kim Noseong Park Jinho Lee Jinho Lee MQ 40 1 0 29 Jul 2024
Depth-Wise Convolutions in Vision Transformers for Efficient Training on Small Datasets Tianxiao Zhang Wenju Xu Bo Luo Guanghui Wang ViT MDE 36 7 0 28 Jul 2024
Continual Distillation Learning: Knowledge Distillation in Prompt-based Continual Learning Qifan Zhang Yunhui Guo Yu Xiang VLM CLL 54 0 0 18 Jul 2024
Wicked Oddities: Selectively Poisoning for Effective Clean-Label Backdoor Attacks Quang H. Nguyen Nguyen Ngoc-Hieu The-Anh Ta Thanh Nguyen-Tang Kok-Seng Wong Hoang Thanh-Tung Khoa D. Doan AAML 29 2 0 15 Jul 2024
Background Adaptation with Residual Modeling for Exemplar-Free Class-Incremental Semantic Segmentation Anqi Zhang Guangyu Gao CLL VLM 33 4 0 13 Jul 2024
Neural-based Video Compression on Solar Dynamics Observatory Images Atefeh Khoshkhahtinat Ali Zafari P. Mehta Nasser M. Nasrabadi Barbara J. Thompson M. Kirk D. D. Silva 44 0 0 12 Jul 2024
On the Role of Discrete Tokenization in Visual Representation Learning Tianqi Du Yifei Wang Yisen Wang 42 7 0 12 Jul 2024
HDKD: Hybrid Data-Efficient Knowledge Distillation Network for Medical Image Classification Omar S. El-Assiouti Ghada Hamed Dina Khattab H. M. Ebied 27 1 0 10 Jul 2024
MambaVision: A Hybrid Mamba-Transformer Vision Backbone Ali Hatamizadeh Jan Kautz Mamba 38 56 0 10 Jul 2024
Fish-Vista: A Multi-Purpose Dataset for Understanding & Identification of Traits from Images Kazi Sajeed Mehrab M. Maruf Arka Daw Harish Babu Manogaran Abhilash Neog ... Paula Mabee Wasila Dahdul Anuj Karpatne Wasila M Dahdul Anuj Karpatne 33 4 0 10 Jul 2024
Mobile Edge Intelligence for Large Language Models: A Contemporary Survey Guanqiao Qu Qiyuan Chen Wei Wei Zheng Lin Xianhao Chen Kaibin Huang 40 43 0 09 Jul 2024
HiT-SR: Hierarchical Transformer for Efficient Image Super-Resolution Xiang Zhang Yulun Zhang Fisher Yu 37 15 0 08 Jul 2024
Learning Motion Blur Robust Vision Transformers with Dynamic Early Exit for Real-Time UAV Tracking You Wu Xucheng Wang Dan Zeng Hengzhou Ye Xiaolan Xie Qijun Zhao Shuiwang Li 29 3 0 07 Jul 2024
Isomorphic Pruning for Vision Models Gongfan Fang Xinyin Ma Michael Bi Mi Xinchao Wang VLM ViT 34 6 0 05 Jul 2024
LPViT: Low-Power Semi-structured Pruning for Vision Transformers Kaixin Xu Zhe Wang Chunyun Chen Xue Geng Jie Lin Xulei Yang Min-man Wu Min Wu Xiaoli Li Weisi Lin ViT VLM 43 6 0 02 Jul 2024
A Sanity Check for AI-generated Image Detection Shilin Yan Ouxiang Li Jiayin Cai Y. Hao Xiaolong Jiang Yao Hu Weidi Xie VLM 64 20 0 27 Jun 2024
Mind the Graph When Balancing Data for Fairness or Robustness Jessica Schrouff Alexis Bellot Amal Rannen-Triki Alan Malek Isabela Albuquerque A. Gretton Alexander DÁmour Silvia Chiappa OOD CML 26 1 0 25 Jun 2024
LOGCAN++: Adaptive Local-global class-aware network for semantic segmentation of remote sensing imagery Xiaowen Ma Rongrong Lian Zhenkai Wu Hongbo Guo Mengting Ma Sensen Wu Zhenhong Du Siyang Song Wei Zhang 39 4 0 24 Jun 2024
Deciphering the Definition of Adversarial Robustness for post-hoc OOD Detectors Peter Lorenz Mario Fernandez Jens Müller Ullrich Kothe AAML 73 1 0 21 Jun 2024
CT3D++: Improving 3D Object Detection with Keypoint-induced Channel-wise Transformer Hualian Sheng Sijia Cai Na Zhao Bing Deng Qiao Liang Min-Jian Zhao Jieping Ye 3DPC 40 0 0 12 Jun 2024
Adaptively Bypassing Vision Transformer Blocks for Efficient Visual Tracking Xiangyang Yang Dan Zeng Xucheng Wang You Wu Hengzhou Ye Qijun Zhao Shuiwang Li 53 3 0 12 Jun 2024
Particle Multi-Axis Transformer for Jet Tagging Muhammad Usman M. Shahid Maheen Ejaz Ummay Hani Nayab Fatima Abdul Rehman Khan Asifullah Khan Nasir Majid Mirza 24 3 0 09 Jun 2024
DIRECT-3D: Learning Direct Text-to-3D Generation on Massive Noisy 3D Data Qihao Liu Yi Zhang Song Bai Adam Kortylewski Alan Yuille 34 9 0 06 Jun 2024
ReDistill: Residual Encoded Distillation for Peak Memory Reduction of CNNs Fang Chen Gourav Datta Mujahid Al Rafi Hyeran Jeon Meng Tang 91 1 0 06 Jun 2024
The 3D-PC: a benchmark for visual perspective taking in humans and machines Drew Linsley Peisen Zhou A. Ashok Akash Nagaraj Gaurav Gaonkar Francis E Lewis Zygmunt Pizlo Thomas Serre 46 6 0 06 Jun 2024
Language-guided Detection and Mitigation of Unknown Dataset Bias Zaiying Zhao Soichiro Kumano Toshihiko Yamasaki 34 2 0 05 Jun 2024
Tiny models from tiny data: Textual and null-text inversion for few-shot distillation Erik Landolsi Fredrik Kahl DiffM 53 1 0 05 Jun 2024
Scaling White-Box Transformers for Vision Jinrui Yang Xianhang Li Druv Pai Yuyin Zhou Yi-An Ma Yaodong Yu Cihang Xie ViT 34 9 0 30 May 2024
Sharing Key Semantics in Transformer Makes Efficient Image Restoration Bin Ren Yawei Li Jingyun Liang Rakesh Ranjan Mengyuan Liu Rita Cucchiara Luc Van Gool Ming-Hsuan Yang N. Sebe 30 3 0 30 May 2024
P $^2$ -ViT: Power-of-Two Post-Training Quantization and Acceleration for Fully Quantized Vision Transformer Huihong Shi Xin Cheng Wendong Mao Zhongfeng Wang MQ 40 3 0 30 May 2024
Don't drop your samples! Coherence-aware training benefits Conditional diffusion Nicolas Dufour Victor Besnier Vicky Kalogeiton David Picard DiffM 49 2 0 30 May 2024