Training data-efficient image transformers & distillation through attention

23 December 2020

Alexandre Sablayrolles

Hervé Jégou

ViT

ArXiv PDF HTML

Papers citing "Training data-efficient image transformers & distillation through attention"

50 / 1,051 papers shown

Title
EViT: An Eagle Vision Transformer with Bi-Fovea Self-Attention Yulong Shi Mingwei Sun Yongshuai Wang Hui Sun Zengqiang Chen 29 4 0 10 Oct 2023
Conformal Prediction for Deep Classifier via Label Ranking Jianguo Huang Huajun Xi Linjun Zhang Huaxiu Yao Yue Qiu Hongxin Wei 33 21 0 10 Oct 2023
CoinSeg: Contrast Inter- and Intra- Class Representations for Incremental Segmentation Zekang Zhang Guangyu Gao Jianbo Jiao C. Liu Yunchao Wei 46 21 0 10 Oct 2023
Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity Lu Yin You Wu Zhenyu (Allen) Zhang Cheng-Yu Hsieh Yaqing Wang ... Mykola Pechenizkiy Yi Liang Michael Bendersky Zhangyang Wang Shiwei Liu 28 78 0 08 Oct 2023
Low-Resolution Self-Attention for Semantic Segmentation Yu-Huan Wu Shi-Chen Zhang Yun-Hai Liu Le Zhang Xin Zhan Daquan Zhou Jiashi Feng Ming-Ming Cheng Liangli Zhen ViT 32 3 0 08 Oct 2023
NOLA: Compressing LoRA using Linear Combination of Random Basis Soroush Abbasi Koohpayegani K. Navaneet Parsa Nooralinejad Soheil Kolouri Hamed Pirsiavash 35 12 0 04 Oct 2023
Efficient Supervised Training of Audio Transformers for Music Representation Learning Pablo Alonso-Jiménez Xavier Serra Dmitry Bogdanov ViT 27 3 0 28 Sep 2023
Weight Averaging Improves Knowledge Distillation under Domain Shift Valeriy Berezovskiy Nikita Morozov MoMe 19 1 0 20 Sep 2023
Interpretability-Aware Vision Transformer Yao Qiang Chengyin Li Prashant Khanduri D. Zhu ViT 80 7 0 14 Sep 2023
3D Transformer based on deformable patch location for differential diagnosis between Alzheimer's disease and Frontotemporal dementia H. Nguyen Michael Clement Boris Mansencal Pierrick Coupé MedIm 26 0 0 06 Sep 2023
Prototype-based Dataset Comparison N. V. Noord 23 6 0 05 Sep 2023
AGS: An Dataset and Taxonomy for Domestic Scene Sound Event Recognition Nan Che Chenrui Liu Fei Yu 25 0 0 30 Aug 2023
Uncovering the Hidden Cost of Model Compression Diganta Misra Muawiz Chaudhary Agam Goyal Bharat Runwal Pin-Yu Chen VLM 24 0 0 29 Aug 2023
PanoSwin: a Pano-style Swin Transformer for Panorama Understanding Zhixin Ling Zhen Xing Xiangdong Zhou Manliang Cao G. Zhou ViT 21 17 0 28 Aug 2023
With a Little Help from your own Past: Prototypical Memory Networks for Image Captioning Manuele Barraco Sara Sarto Marcella Cornia Lorenzo Baraldi Rita Cucchiara VLM 51 19 0 23 Aug 2023
TurboViT: Generating Fast Vision Transformers via Generative Architecture Search Alexander Wong Saad Abbasi Saeejith Nair ViT 25 1 0 22 Aug 2023
UnLoc: A Unified Framework for Video Localization Tasks Shengjia Yan Xuehan Xiong Arsha Nagrani Anurag Arnab Zhonghao Wang Weina Ge David A. Ross Cordelia Schmid 22 53 0 21 Aug 2023
MGMAE: Motion Guided Masking for Video Masked Autoencoding Bingkun Huang Zhiyu Zhao Guozhen Zhang Yu Qiao Limin Wang 22 30 0 21 Aug 2023
CoNe: Contrast Your Neighbours for Supervised Image Classification Mingkai Zheng Shan You Lang Huang Xiu Su Fei Wang Chao Qian Xiaogang Wang Chang Xu VLM 20 0 0 21 Aug 2023
HODN: Disentangling Human-Object Feature for HOI Detection Shuman Fang Zhiwen Lin Ke Yan Jie Li Xianming Lin Rongrong Ji 44 5 0 20 Aug 2023
Which Transformer to Favor: A Comparative Analysis of Efficiency in Vision Transformers Tobias Christian Nauen Sebastián M. Palacio Federico Raue Andreas Dengel 37 3 0 18 Aug 2023
AudioFormer: Audio Transformer learns audio feature representations from discrete acoustic codes Zhaohui Li Haitao Wang Xinghua Jiang 31 1 0 14 Aug 2023
Advancing Natural-Language Based Audio Retrieval with PaSST and Large Audio-Caption Data Sets Paul Primus Khaled Koutini Gerhard Widmer 19 13 0 08 Aug 2023
SSTFormer: Bridging Spiking Neural Network and Memory Support Transformer for Frame-Event based Recognition Xiao Wang Zong-Yao Wu Yao Rong Lin Zhu Bowei Jiang Jin Tang Yonghong Tian ViT 69 14 0 08 Aug 2023
FLIQS: One-Shot Mixed-Precision Floating-Point and Integer Quantization Search Jordan Dotzel Gang Wu Andrew Li M. Umar Yun Ni ... Liqun Cheng Martin G. Dixon N. Jouppi Quoc V. Le Sheng R. Li MQ 25 3 0 07 Aug 2023
DETR Doesn't Need Multi-Scale or Locality Design Yutong Lin Yuhui Yuan Zheng-Wei Zhang Chen Li Nanning Zheng Han Hu 30 5 0 03 Aug 2023
Deep Learning and Computer Vision for Glaucoma Detection: A Review Mona Ashtari-Majlan Mohammad Mahdi Dehshibi David Masip 25 9 0 31 Jul 2023
BARTPhoBEiT: Pre-trained Sequence-to-Sequence and Image Transformers Models for Vietnamese Visual Question Answering Khiem Vinh Tran Kiet Van Nguyen N. Nguyen ViT 23 2 0 28 Jul 2023
Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models Dong Lu Zhiqiang Wang Teng Wang Weili Guan Hongchang Gao Feng Zheng AAML 46 65 0 26 Jul 2023
Visual Prompt Flexible-Modal Face Anti-Spoofing Zitong Yu Rizhao Cai Yawen Cui Ajian Liu Changsheng Chen 32 6 0 26 Jul 2023
Foundational Models Defining a New Era in Vision: A Survey and Outlook Muhammad Awais Muzammal Naseer Salman Khan Rao Muhammad Anwer Hisham Cholakkal M. Shah Ming Yang F. Khan VLM 24 118 0 25 Jul 2023
Quantized Feature Distillation for Network Quantization Kevin Zhu Yin He Jianxin Wu MQ 24 9 0 20 Jul 2023
Improving Domain Generalization for Sound Classification with Sparse Frequency-Regularized Transformer Honglin Mu Wentian Xia Wanxiang Che 10 1 0 19 Jul 2023
Hierarchical Spatiotemporal Transformers for Video Object Segmentation Jun-Sang Yoo H. Lee Seung‐Won Jung VOS 26 1 0 17 Jul 2023
Mitigating Adversarial Vulnerability through Causal Parameter Estimation by Adversarial Double Machine Learning Byung-Kwan Lee Junho Kim Yonghyun Ro AAML 10 9 0 14 Jul 2023
Random Position Adversarial Patch for Vision Transformers Mingzhen Shao ViT AAML 16 2 0 09 Jul 2023
All in One: Exploring Unified Vision-Language Tracking with Multi-Modal Alignment Chunhui Zhang Xin Sun Li Liu Yiqian Yang Qiong Liu Xiaoping Zhou Yanfeng Wang 38 15 0 07 Jul 2023
Spike-driven Transformer Man Yao Jiakui Hu Zhaokun Zhou Liuliang Yuan Yonghong Tian Boxing Xu Guoqi Li 34 112 0 04 Jul 2023
MobileViG: Graph-Based Sparse Attention for Mobile Vision Applications Mustafa Munir William Avery R. Marculescu ViT GNN 31 33 0 01 Jul 2023
Long-Tailed Continual Learning For Visual Food Recognition Jiangpeng He Luotao Lin Jack Ma H. Eicher-Miller F. Zhu Fengqing M Zhu 59 14 0 01 Jul 2023
Cross Architecture Distillation for Face Recognition Weisong Zhao Xiangyu Zhu Zhixiang He Xiaoyu Zhang Zhen Lei CVBM 15 6 0 26 Jun 2023
A Simple and Effective Pruning Approach for Large Language Models Mingjie Sun Zhuang Liu Anna Bair J. Zico Kolter 56 353 0 20 Jun 2023
GIO: Gradient Information Optimization for Training Dataset Selection Dante Everaert Christopher Potts 19 3 0 20 Jun 2023
Stochastic Re-weighted Gradient Descent via Distributionally Robust Optimization Ramnath Kumar Kushal Majmundar Dheeraj M. Nagaraj A. Suggala ODL 24 6 0 15 Jun 2023
Instruct-ReID: A Multi-purpose Person Re-identification Task with Instructions Weizhen He Yihe Deng Shixiang Tang Qihao Chen Qingsong Xie ... Feng Zhu Rui Zhao Wanli Ouyang Donglian Qi Yunfeng Yan 65 19 0 13 Jun 2023
InvPT++: Inverted Pyramid Multi-Task Transformer for Visual Scene Understanding Hanrong Ye Dan Xu ViT 23 10 0 08 Jun 2023
Energy-Based Models for Cross-Modal Localization using Convolutional Transformers Alan Wu Michael S. Ryoo 25 3 0 06 Jun 2023
Performance-optimized deep neural networks are evolving into worse models of inferotemporal visual cortex Drew Linsley I. F. Rodriguez Thomas Fel Michael Arcaro Saloni Sharma Margaret Livingstone Thomas Serre 22 18 0 06 Jun 2023
Quantifying the Variability Collapse of Neural Networks Jing-Xue Xu Haoxiong Liu 31 4 0 06 Jun 2023
Centered Self-Attention Layers Ameen Ali Tomer Galanti Lior Wolf 28 6 0 02 Jun 2023