Training data-efficient image transformers & distillation through attention

23 December 2020

Alexandre Sablayrolles

Hervé Jégou

ViT

ArXiv PDF HTML

Papers citing "Training data-efficient image transformers & distillation through attention"

50 / 983 papers shown

Title
Parallel Sequence Modeling via Generalized Spatial Propagation Network Hongjun Wang Wonmin Byeon Jiarui Xu Jinwei Gu Ka Chun Cheung Xiaolong Wang Kai Han Jan Kautz Sifei Liu 117 0 0 21 Jan 2025
UAV-Assisted Real-Time Disaster Detection Using Optimized Transformer Model Branislava Jankovic Sabina Jangirova Waseem Ullah Latif U. Khan Mohsen Guizani 31 0 0 21 Jan 2025
IncSAR: A Dual Fusion Incremental Learning Framework for SAR Target Recognition George Karantaidis Athanasios Pantsios Y. Kompatsiaris Symeon Papadopoulos CLL 60 0 0 20 Jan 2025
Elucidating the Design Space of Dataset Condensation Shitong Shao Zikai Zhou Huanran Chen Zhiqiang Shen DD 54 7 0 20 Jan 2025
MoRe: Class Patch Attention Needs Regularization for Weakly Supervised Semantic Segmentation Zhiwei Yang Yucong Meng Kexue Fu Shuo Wang Zhijian Song 85 1 0 20 Jan 2025
Geometric Distortion Guided Transformer for Omnidirectional Image Super-Resolution Cuixin Yang Rongkang Dong Jun Xiao Cong Zhang Kin-Man Lam Fei Zhou Guoping Qiu 81 1 0 17 Jan 2025
Protego: Detecting Adversarial Examples for Vision Transformers via Intrinsic Capabilities Jialin Wu Kaikai Pan Yanjiao Chen Jiangyi Deng Shengyuan Pang Wenyuan Xu ViT AAML 43 0 0 13 Jan 2025
SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Training Tianjin Huang Ziquan Zhu Gaojie Jin Lu Liu Zhangyang Wang Shiwei Liu 42 1 0 12 Jan 2025
MS-Temba : Multi-Scale Temporal Mamba for Efficient Temporal Action Detection Arkaprava Sinha Monish Soundar Raj Pu Wang Ahmed Helmy Srijan Das Mamba 53 3 0 10 Jan 2025
Multiscaled Multi-Head Attention-based Video Transformer Network for Hand Gesture Recognition Mallika Garg Debashis Ghosh P. M. Pradhan SLR 30 16 0 03 Jan 2025
Keypoint Aware Masked Image Modelling Madhava Krishna Convin.AI 65 0 0 03 Jan 2025
Advancements in Visual Language Models for Remote Sensing: Datasets, Capabilities, and Enhancement Techniques Lijie Tao H. Zhang Haizhao Jing Yu Liu Kelu Yao Guoting Wei Xizhe Xue 33 0 0 03 Jan 2025
Semantics Prompting Data-Free Quantization for Low-Bit Vision Transformers Yunshan Zhong Yuyao Zhou Yuxin Zhang Shen Li Yong Li Fei Chao Zhanpeng Zeng Rongrong Ji MQ 94 0 0 31 Dec 2024
VMamba: Visual State Space Model Yue Liu Yunjie Tian Yuzhong Zhao Hongtian Yu Lingxi Xie Yaowei Wang Qixiang Ye Jianbin Jiao Yunfan Liu Mamba 113 609 0 31 Dec 2024
Uncertainty Herding: One Active Learning Method for All Label Budgets Wonho Bae Gabriel L. Oliveira Danica J. Sutherland UQCV 124 0 0 30 Dec 2024
SeagrassFinder: Deep Learning for Eelgrass Detection and Coverage Estimation in the Wild Jannik Elsäßer Laura Weihl Veronika Cheplygina Lisbeth Tangaa Nielsen 74 0 0 20 Dec 2024
GG-SSMs: Graph-Generating State Space Models Nikola Zubić Davide Scaramuzza Mamba 88 1 0 17 Dec 2024
Video Diffusion Transformers are In-Context Learners Zhengcong Fei Di Qiu Changqian Yu Debang Li Mingyuan Fan VGen DiffM 169 2 0 14 Dec 2024
Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMs Qizhe Zhang Aosong Cheng Ming Lu Zhiyong Zhuo Minqi Wang Jiajun Cao Shaobo Guo Qi She Shanghang Zhang VLM 88 11 0 02 Dec 2024
Training Noise Token Pruning Mingxing Rao Bohan Jiang Daniel Moyer ViT 72 0 0 27 Nov 2024
COAP: Memory-Efficient Training with Correlation-Aware Gradient Projection Jinqi Xiao S. Sang Tiancheng Zhi Jing Liu Qing Yan Linjie Luo Bo Yuan Bo Yuan VLM 83 1 0 26 Nov 2024
EfficientViM: Efficient Vision Mamba with Hidden State Mixer based State Space Duality Sanghyeok Lee Joonmyung Choi Hyunwoo J. Kim 110 3 0 22 Nov 2024
Principles of Visual Tokens for Efficient Video Understanding Xinyue Hao Gen Li Shreyank N. Gowda Robert B Fisher Jonathan Huang Anurag Arnab Laura Sevilla-Lara 92 0 0 20 Nov 2024
Breaking the Low-Rank Dilemma of Linear Attention Qihang Fan Huaibo Huang Ran He 33 0 0 12 Nov 2024
DiMSUM: Diffusion Mamba -- A Scalable and Unified Spatial-Frequency Method for Image Generation Hao Phung Quan Dao T. Dao Hoang Phan Dimitris Metaxas Anh Tran Mamba 62 3 0 06 Nov 2024
MambaPEFT: Exploring Parameter-Efficient Fine-Tuning for Mamba Masakazu Yoshimura Teruaki Hayashi Yota Maeda Mamba 87 2 0 06 Nov 2024
On Improved Conditioning Mechanisms and Pre-training Strategies for Diffusion Models Tariq Berrada Ifriqi Pietro Astolfi Melissa Hall Reyhane Askari Hemmat Yohann Benchetrit ... Matthew Muckley Karteek Alahari Adriana Romero Soriano Jakob Verbeek M. Drozdzal AI4CE VLM 52 2 0 05 Nov 2024
ViT-LCA: A Neuromorphic Approach for Vision Transformers Sanaz Mahmoodi Takaghaj ViT 45 0 0 31 Oct 2024
Driving by the Rules: A Benchmark for Integrating Traffic Sign Regulations into Vectorized HD Map Xinyuan Chang Maixuan Xue Xinran Liu Zheng Pan Xing Wei 40 1 0 31 Oct 2024
TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters Haiyang Wang Yue Fan Muhammad Ferjad Naeem Yongqin Xian J. E. Lenssen Liwei Wang F. Tombari Bernt Schiele 41 2 0 30 Oct 2024
PViT: Prior-augmented Vision Transformer for Out-of-distribution Detection Tianhao Zhang Zhixiang Chen Lyudmila Mihaylova 92 0 0 27 Oct 2024
ViMoE: An Empirical Study of Designing Vision Mixture-of-Experts Xumeng Han Longhui Wei Zhiyang Dou Zipeng Wang Chenhui Qiang Xin He Yingfei Sun Zhenjun Han Qi Tian MoE 37 3 0 21 Oct 2024
Spatial-Mamba: Effective Visual State Space Models via Structure-aware State Fusion Chaodong Xiao Minghan Li Zhengqiang Zhang Deyu Meng Lei Zhang Mamba 55 4 0 19 Oct 2024
MomentumSMoE: Integrating Momentum into Sparse Mixture of Experts R. Teo Tan M. Nguyen MoE 31 3 0 18 Oct 2024
Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learning Yuxiang Lu Shengcao Cao Yu-xiong Wang 45 1 0 18 Oct 2024
MoH: Multi-Head Attention as Mixture-of-Head Attention Peng Jin Bo Zhu Li Yuan Shuicheng Yan MoE 29 13 0 15 Oct 2024
InvSeg: Test-Time Prompt Inversion for Semantic Segmentation Jiayi Lin Jiabo Huang Jian Hu S. Gong DiffM VLM 30 0 0 15 Oct 2024
GlobalMamba: Global Image Serialization for Vision Mamba Chengkun Wang Wenzhao Zheng Jie Zhou Jiwen Lu Mamba 31 0 0 14 Oct 2024
S $^4$ ST: A Strong, Self-transferable, faSt, and Simple Scale Transformation for Transferable Targeted Attack Yongxiang Liu Bowen Peng Li Liu X. Li 84 0 0 13 Oct 2024
M $^2$ -ViT: Accelerating Hybrid Vision Transformers with Two-Level Mixed Quantization Yanbiao Liang Huihong Shi Zhongfeng Wang MQ 21 0 0 10 Oct 2024
SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration Heming Xia Yongqi Li Jun Zhang Cunxiao Du Wenjie Li LRM 46 5 0 09 Oct 2024
Compositional Entailment Learning for Hyperbolic Vision-Language Models Avik Pal Max van Spengler Guido Maria DÁmely di Melendugno Alessandro Flaborea Fabio Galasso Pascal Mettes CoGe 40 5 0 09 Oct 2024
Learning the Generalizable Manipulation Skills on Soft-body Tasks via Guided Self-attention Behavior Cloning Policy XueTao Li Fang Gao Jun Yu Shaodong Li Feng Shuang LM&Ro 18 0 0 08 Oct 2024
Designing Concise ConvNets with Columnar Stages Ashish Kumar Jaesik Park MQ 23 0 0 05 Oct 2024
Dynamic Sparse Training versus Dense Training: The Unexpected Winner in Image Corruption Robustness Boqian Wu Q. Xiao Shunxin Wang N. Strisciuglio Mykola Pechenizkiy M. V. Keulen D. Mocanu Elena Mocanu OOD 3DH 52 0 0 03 Oct 2024
Theoretical Insights into Fine-Tuning Attention Mechanism: Generalization and Optimization Xinhao Yao Hongjin Qian Xiaolin Hu Gengze Xu Wei Liu Jian Luan B. Wang Y. Liu 48 0 0 03 Oct 2024
MAP: Unleashing Hybrid Mamba-Transformer Vision Backbone's Potential with Masked Autoregressive Pretraining Yunze Liu Li Yi Mamba 45 2 0 01 Oct 2024
Multi-View and Multi-Scale Alignment for Contrastive Language-Image Pre-training in Mammography Yuexi Du John Onofrey Nicha Dvornek VLM 45 1 0 26 Sep 2024
AI-driven View Guidance System in Intra-cardiac Echocardiography Imaging Jaeyoung Huh Paul Klein Gareth Funka-Lea Puneet Sharma A. Kapoor Young-Ho Kim 36 3 0 25 Sep 2024
OmniBench: Towards The Future of Universal Omni-Language Models Yizhi Li Ge Zhang Yinghao Ma Ruibin Yuan Kang Zhu ... Zhaoxiang Zhang Zachary Liu Emmanouil Benetos Wenhao Huang Chenghua Lin LRM 44 11 0 23 Sep 2024