Training data-efficient image transformers & distillation through attention

23 December 2020

Alexandre Sablayrolles

Hervé Jégou

ViT

ArXiv PDF HTML

Papers citing "Training data-efficient image transformers & distillation through attention"

50 / 983 papers shown

Title
Instruct-ReID++: Towards Universal Purpose Instruction-Guided Person Re-identification Weizhen He Yiheng Deng Yunfeng Yan Feng Zhu Yizhou Wang Lei Bai Qingsong Xie Donglian Qi Wanli Ouyang Shixiang Tang 93 2 0 28 May 2024
Exploiting the Layered Intrinsic Dimensionality of Deep Models for Practical Adversarial Training Enes Altinisik Safa Messaoud H. Sencar Hassan Sajjad Sanjay Chawla AAML 43 0 0 27 May 2024
Understanding the Effect of using Semantically Meaningful Tokens for Visual Representation Learning N. Kalibhat Priyatham Kattakinda Arman Zarei Nikita Seleznev Sam Sharpe Senthil Kumar S. Feizi ViT 34 0 0 26 May 2024
Accelerating Transformers with Spectrum-Preserving Token Merging Hoai-Chau Tran D. M. Nguyen Duy M. Nguyen Trung Thanh Nguyen Ngan Le Pengtao Xie Daniel Sonntag James Y. Zou Binh T. Nguyen Mathias Niepert 32 8 0 25 May 2024
Dinomaly: The Less Is More Philosophy in Multi-Class Unsupervised Anomaly Detection Jia Guo Shuai Lu Weihang Zhang Huiqi Li Huiqi Li Hongen Liao ViT 59 7 0 23 May 2024
LookHere: Vision Transformers with Directed Attention Generalize and Extrapolate A. Fuller Daniel G. Kyrollos Yousef Yassin James R. Green 34 2 0 22 May 2024
Vision Transformer with Sparse Scan Prior Qihang Fan Huaibo Huang Mingrui Chen Ran He ViT 36 5 0 22 May 2024
SLAB: Efficient Transformers with Simplified Linear Attention and Progressive Re-parameterized Batch Normalization Jialong Guo Xinghao Chen Yehui Tang Yunhe Wang ViT 47 9 0 19 May 2024
$${M^2D}$NeRF: Multi-Modal Decomposition NeRF with 3D Feature Fields$ ${M^2D}$ NeRF: Multi-Modal Decomposition NeRF with 3D Feature Fields N. Wang Lefei Zhang Angel X Chang 44 0 0 08 May 2024
S3Former: Self-supervised High-resolution Transformer for Solar PV Profiling Minh-Triet Tran Adrian de Luis Haitao Liao Ying Huang Roy McCann Alan Mantooth Jack Cothren Ngan Le 82 0 0 07 May 2024
Key Patches Are All You Need: A Multiple Instance Learning Framework For Robust Medical Diagnosis Diogo J. Araújo M. R. Verdelho Alceu Bissoto Jacinto C. Nascimento Carlos Santiago Catarina Barata 27 1 0 02 May 2024
ShadowMaskFormer: Mask Augmented Patch Embeddings for Shadow Removal Zhuohao Li Guoyang Xie Guannan Jiang Zhichao Lu 31 3 0 29 Apr 2024
Embracing Diversity: Interpretable Zero-shot classification beyond one vector per class Mazda Moayeri Michael G. Rabbat Mark Ibrahim Diane Bouchacourt VLM 41 1 0 25 Apr 2024
Understanding Hyperbolic Metric Learning through Hard Negative Sampling Yun Yue Fangzhou Lin Guanyi Mou Ziming Zhang SSL 30 1 0 23 Apr 2024
StrideNET: Swin Transformer for Terrain Recognition with Dynamic Roughness Extraction Maitreya Shelare Neha Shigvan Atharva Satam Poonam Sonar 14 1 0 20 Apr 2024
AMU-Tuning: Effective Logit Bias for CLIP-based Few-shot Learning Yuwei Tang Zhenyi Lin Qilong Wang Pengfei Zhu Qinghua Hu 26 11 0 13 Apr 2024
DWE+: Dual-Way Matching Enhanced Framework for Multimodal Entity Linking Shezheng Song Shasha Li Shan Zhao Xiaopeng Li Chengyu Wang Jie Yu Jun Ma Tianwei Yan Bing Ji Xiaoguang Mao 23 0 0 07 Apr 2024
A Multi-Level Framework for Accelerating Training Transformer Models Longwei Zou Han Zhang Yangdong Deng AI4CE 32 1 0 07 Apr 2024
Dissecting Query-Key Interaction in Vision Transformers Xu Pan Aaron Philip Ziqian Xie Odelia Schwartz 32 1 0 04 Apr 2024
Instance-Aware Group Quantization for Vision Transformers Jaehyeon Moon Dohyung Kim Junyong Cheon Bumsub Ham MQ ViT 27 5 0 01 Apr 2024
Training-Free Semantic Segmentation via LLM-Supervision Wenfang Sun Yingjun Du Gaowen Liu Ramana Rao Kompella Cees G. M. Snoek VLM 35 2 0 31 Mar 2024
SpiralMLP: A Lightweight Vision MLP Architecture Haojie Mu Burhan Ul Tayyab Nicholas Chua 35 0 0 31 Mar 2024
FreeSeg-Diff: Training-Free Open-Vocabulary Segmentation with Diffusion Models Barbara Toniella Corradini Mustafa Shukor Paul Couairon Guillaume Couairon Franco Scarselli Matthieu Cord DiffM VLM 38 4 0 29 Mar 2024
Efficient Modulation for Vision Networks Xu Ma Xiyang Dai Jianwei Yang Bin Xiao Yinpeng Chen Yun Fu Lu Yuan 38 17 0 29 Mar 2024
Enhancing Efficiency in Vision Transformer Networks: Design Techniques and Insights Moein Heidari Reza Azad Sina Ghorbani Kolahi René Arimond Leon Niggemeier ... Afshin Bozorgpour Ehsan Khodapanah Aghdam A. Kazerouni I. Hacihaliloglu Dorit Merhof 41 7 0 28 Mar 2024
Surgical-LVLM: Learning to Adapt Large Vision-Language Model for Grounded Visual Question Answering in Robotic Surgery Guan-Feng Wang Long Bai Wan Jun Nah Jie Wang Zhaoxi Zhang Zhen Chen Jinlin Wu Mobarakol Islam Hongbin Liu Hongliang Ren 40 14 0 22 Mar 2024
ADAPT to Robustify Prompt Tuning Vision Transformers Masih Eskandar Tooba Imtiaz Zifeng Wang Jennifer Dy VPVLM VLM AAML 36 0 0 19 Mar 2024
HIRI-ViT: Scaling Vision Transformer with High Resolution Inputs Ting Yao Yehao Li Yingwei Pan Tao Mei ViT 23 15 0 18 Mar 2024
A Decade's Battle on Dataset Bias: Are We There Yet? Zhuang Liu Kaiming He 40 28 0 13 Mar 2024
Segmentation Guided Sparse Transformer for Under-Display Camera Image Restoration Jingyun Xue Tao Wang Jun Wang Kaihao Zhang ViT 43 2 0 09 Mar 2024
LORS: Low-rank Residual Structure for Parameter-Efficient Network Stacking Jialin Li Qiang Nie Weifu Fu Yuhuan Lin Guangpin Tao Yong-Jin Liu Chengjie Wang 25 4 0 07 Mar 2024
Domain-Agnostic Mutual Prompting for Unsupervised Domain Adaptation Zhekai Du Xinyao Li Fengling Li Ke Lu Lei Zhu Jingjing Li 38 15 0 05 Mar 2024
Vision-RWKV: Efficient and Scalable Visual Perception with RWKV-Like Architectures Yuchen Duan Weiyun Wang Zhe Chen Xizhou Zhu Lewei Lu Tong Lu Yu Qiao Hongsheng Li Jifeng Dai Wenhai Wang ViT 38 44 0 04 Mar 2024
Large Convolutional Model Tuning via Filter Subspace Wei Chen Zichen Miao Qiang Qiu 49 3 0 01 Mar 2024
Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models Frederik Kunstner Robin Yadav Alan Milligan Mark Schmidt Alberto Bietti 29 26 0 29 Feb 2024
Attention-aware Semantic Communications for Collaborative Inference Jiwoong Im Nayoung Kwon Taewoo Park Jiheon Woo Jaeho Lee Yongjune Kim 41 2 0 23 Feb 2024
FViT: A Focal Vision Transformer with Gabor Filter Yulong Shi Mingwei Sun Yongshuai Wang Rui Wang 47 4 0 17 Feb 2024
Learning Low-Rank Feature for Thorax Disease Classification Rajeev Goel Utkarsh Nath Yancheng Wang Alvin C. Silva Teresa Wu Yingzhen Yang 20 0 0 14 Feb 2024
TransAxx: Efficient Transformers with Approximate Computing Dimitrios Danopoulos Georgios Zervakis Dimitrios Soudris Jörg Henkel ViT 42 2 0 12 Feb 2024
Text-to-Code Generation with Modality-relative Pre-training Fenia Christopoulou Guchun Zhang Gerasimos Lampouras AI4TS 13 1 0 08 Feb 2024
RepQuant: Towards Accurate Post-Training Quantization of Large Transformer Models via Scale Reparameterization Zhikai Li Xuewen Liu Jing Zhang Qingyi Gu MQ 37 7 0 08 Feb 2024
Implicit Bias and Fast Convergence Rates for Self-attention Bhavya Vasudeva Puneesh Deora Christos Thrampoulidis 24 13 0 08 Feb 2024
Parameter-Efficient Fine-Tuning for Pre-Trained Vision Models: A Survey Yi Xin Jianjiang Yang Haodi Zhou Junlong Du Junlong Du Yue Fan Qing Li Qing Li Yuntao Du VLM 59 75 0 03 Feb 2024
CascadedGaze: Efficiency in Global Context Extraction for Image Restoration Amirhosein Ghasemabadi Muhammad Kamran Janjua Mohammad Salameh Chunhua Zhou Fengyu Sun Di Niu 24 11 0 26 Jan 2024
MPTQ-ViT: Mixed-Precision Post-Training Quantization for Vision Transformer Y. Tai An-Yeu Wu Wu MQ 26 6 0 26 Jan 2024
Producing Plankton Classifiers that are Robust to Dataset Shift Cheng Chen S. Kyathanahally Marta Reyes Stefanie Merkli E. Merz Emanuele Francazi Marvin Hoege F. Pomati M. Baity-Jesi 16 2 0 25 Jan 2024
Sparse and Transferable Universal Singular Vectors Attack Kseniia Kuvshinova Olga Tsymboi Ivan V. Oseledets AAML 22 0 0 25 Jan 2024
OnDev-LCT: On-Device Lightweight Convolutional Transformers towards federated learning Chu Myaet Thwal Minh N. H. Nguyen Ye Lin Tun Seongjin Kim My T. Thai Choong Seon Hong 49 5 0 22 Jan 2024
Transformer-CNN Fused Architecture for Enhanced Skin Lesion Segmentation Siddharth Tiwari MedIm ViT 28 0 0 10 Jan 2024
Setting the Record Straight on Transformer Oversmoothing G. Dovonon M. Bronstein Matt J. Kusner 20 5 0 09 Jan 2024