Training data-efficient image transformers & distillation through attention

23 December 2020

Alexandre Sablayrolles

Hervé Jégou

ViT

ArXiv PDF HTML

Papers citing "Training data-efficient image transformers & distillation through attention"

50 / 983 papers shown

Title
Morphing Tokens Draw Strong Masked Image Models Taekyung Kim Byeongho Heo Dongyoon Han 44 3 0 30 Dec 2023
An Empirical Study of Scaling Law for OCR Miao Rang Zhenni Bi Chuanjian Liu Yunhe Wang Kai Han 30 6 0 29 Dec 2023
Adaptive Depth Networks with Skippable Sub-Paths Woochul Kang 28 1 0 27 Dec 2023
Video Recognition in Portrait Mode Mingfei Han Linjie Yang Xiaojie Jin Jiashi Feng Xiaojun Chang Heng Wang 23 3 0 21 Dec 2023
Polynomial-based Self-Attention for Table Representation learning Jayoung Kim Yehjin Shin Jeongwhan Choi Hyowon Wi Noseong Park LMTD 19 2 0 12 Dec 2023
MaxQ: Multi-Axis Query for N:M Sparsity Network Jingyang Xiang Siqi Li Junhao Chen Zhuangzhi Chen Tianxin Huang Linpeng Peng Yong-Jin Liu 16 0 0 12 Dec 2023
MIMIR: Masked Image Modeling for Mutual Information-based Adversarial Robustness Xiaoyun Xu Shujian Yu Jingzheng Wu S. Picek AAML 35 0 0 08 Dec 2023
Graph Convolutions Enrich the Self-Attention in Transformers! Jeongwhan Choi Hyowon Wi Jayoung Kim Yehjin Shin Kookjin Lee Nathaniel Trask Noseong Park 25 4 0 07 Dec 2023
Diversify, Don't Fine-Tune: Scaling Up Visual Recognition Training with Synthetic Images Zhuoran Yu Chenchen Zhu Sean Culatana Raghuraman Krishnamoorthi Fanyi Xiao Yong Jae Lee 109 14 0 04 Dec 2023
SCHEME: Scalable Channel Mixer for Vision Transformers Deepak Sridhar Yunsheng Li Nuno Vasconcelos 18 0 0 01 Dec 2023
Initializing Models with Larger Ones Zhiqiu Xu Yanjie Chen Kirill Vishniakov Yida Yin Zhiqiang Shen Trevor Darrell Lingjie Liu Zhuang Liu 28 17 0 30 Nov 2023
Improving Interpretation Faithfulness for Vision Transformers Lijie Hu Yixin Liu Ninghao Liu Mengdi Huai Lichao Sun Di Wang 21 5 0 29 Nov 2023
PViT-6D: Overclocking Vision Transformers for 6D Pose Estimation with Confidence-Level Prediction and Pose Tokens Sebastian Stapf Tobias Bauernfeind Marco Riboldi ViT 22 1 0 29 Nov 2023
QuadraNet: Improving High-Order Neural Interaction Efficiency with Hardware-Aware Quadratic Neural Networks Chenhui Xu Fuxun Yu Zirui Xu Chenchen Liu Jinjun Xiong Xiang Chen 30 4 0 29 Nov 2023
Efficient Stitchable Task Adaptation Haoyu He Zizheng Pan Jing Liu Jianfei Cai Bohan Zhuang 24 3 0 29 Nov 2023
No Representation Rules Them All in Category Discovery S. Vaze Andrea Vedaldi Andrew Zisserman OOD 26 31 0 28 Nov 2023
Vulnerability Analysis of Transformer-based Optical Character Recognition to Adversarial Attacks Lucas Beerens D. Higham 26 1 0 28 Nov 2023
HEViTPose: High-Efficiency Vision Transformer for Human Pose Estimation Chengpeng Wu Guangxing Tan Chunyu Li ViT 21 0 0 22 Nov 2023
Improving Source-Free Target Adaptation with Vision Transformers Leveraging Domain Representation Images Gauransh Sawhney Daksh Dave Adeel Ahmed Jiechao Gao Khalid Saleem 23 0 0 21 Nov 2023
Improved TokenPose with Sparsity Anning Li ViT 32 0 0 16 Nov 2023
Contrastive Transformer Learning with Proximity Data Generation for Text-Based Person Search Hefeng Wu Weifeng Chen Zhibin Liu Tianshui Chen Zhiguang Chen Liang Lin 28 11 0 15 Nov 2023
4K-Resolution Photo Exposure Correction at 125 FPS with ~8K Parameters Yijie Zhou Chao Li Jin Liang Tianyi Xu Xin Liu Jun Xu 3DV 19 10 0 15 Nov 2023
Learning Contrastive Self-Distillation for Ultra-Fine-Grained Visual Categorization Targeting Limited Samples Ziye Fang Xin Jiang Hao Tang Zechao Li 24 12 0 10 Nov 2023
Dynamic Association Learning of Self-Attention and Convolution in Image Restoration Kui Jiang Xuemei Jia Wenxin Huang Wenbin Wang Zheng Wang Junjun Jiang 20 1 0 09 Nov 2023
Rotation Invariant Transformer for Recognizing Object in UAVs Shuo Chen Mang Ye Bo Du ViT 30 18 0 05 Nov 2023
BarcodeBERT: Transformers for Biodiversity Analysis Pablo Millán Arias Niousha Sadjadi Monireh Safari ZeMing Gong Austin T. Wang ... Iuliia Zarubiieva Dirk Steinke Lila Kari Angel X. Chang Graham W. Taylor 42 7 0 04 Nov 2023
What Makes Pre-Trained Visual Representations Successful for Robust Manipulation? Kaylee Burns Zach Witzel Jubayer Ibn Hamid Tianhe Yu Chelsea Finn Karol Hausman OOD SSL 23 22 0 03 Nov 2023
PAUMER: Patch Pausing Transformer for Semantic Segmentation Evann Courdier Prabhu Teja Sivaprasad F. Fleuret 31 2 0 01 Nov 2023
Mixture-of-Experts for Open Set Domain Adaptation: A Dual-Space Detection Approach Zhenbang Du Jiayu An Yunlu Tu Jiahao Hong Dongrui Wu MoE 20 1 0 01 Nov 2023
OpenForest: A data catalogue for machine learning in forest monitoring Arthur Ouaknine T. Kattenborn Etienne Laliberté David Rolnick 43 5 0 01 Nov 2023
Are Natural Domain Foundation Models Useful for Medical Image Classification? Joana Palés Huix Adithya Raju Ganeshan Johan Fredin Haslum Magnus P Soderberg Christos Matsoukas Kevin Smith OOD MedIm VLM 19 30 0 30 Oct 2023
TransXNet: Learning Both Global and Local Dynamics with a Dual Dynamic Token Mixer for Visual Recognition Meng Lou Hong-Yu Zhou Sibei Yang Yizhou Yu Chuan Wu Yizhou Yu ViT 36 36 0 30 Oct 2023
Towards Generalized Multi-stage Clustering: Multi-view Self-distillation Jiatai Wang Zhiwei Xu Xin Wang Tao Li 11 1 0 29 Oct 2023
Understanding the Effects of Projectors in Knowledge Distillation Yudong Chen Sen Wang Jiajun Liu Xuwei Xu Frank de Hoog Brano Kusy Zi Huang 21 0 0 26 Oct 2023
Auxiliary Features-Guided Super Resolution for Monte Carlo Rendering Qiqi Hou Feng Liu SupR 13 4 0 20 Oct 2023
Minimalist and High-Performance Semantic Segmentation with Plain Vision Transformers Yuanduo Hong Jue Wang Weichao Sun Huihui Pan VLM ViT 32 7 0 19 Oct 2023
Multi-domain improves out-of-distribution and data-limited scenarios for medical image analysis Ece Ozkan Xavier Boix OOD 20 0 0 10 Oct 2023
EViT: An Eagle Vision Transformer with Bi-Fovea Self-Attention Yulong Shi Mingwei Sun Yongshuai Wang Hui Sun Zengqiang Chen 29 4 0 10 Oct 2023
Conformal Prediction for Deep Classifier via Label Ranking Jianguo Huang Huajun Xi Linjun Zhang Huaxiu Yao Yue Qiu Hongxin Wei 31 21 0 10 Oct 2023
CoinSeg: Contrast Inter- and Intra- Class Representations for Incremental Segmentation Zekang Zhang Guangyu Gao Jianbo Jiao C. Liu Yunchao Wei 46 20 0 10 Oct 2023
Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity Lu Yin You Wu Zhenyu (Allen) Zhang Cheng-Yu Hsieh Yaqing Wang ... Mykola Pechenizkiy Yi Liang Michael Bendersky Zhangyang Wang Shiwei Liu 28 78 0 08 Oct 2023
Low-Resolution Self-Attention for Semantic Segmentation Yu-Huan Wu Shi-Chen Zhang Yun-Hai Liu Le Zhang Xin Zhan Daquan Zhou Jiashi Feng Ming-Ming Cheng Liangli Zhen ViT 32 3 0 08 Oct 2023
Efficient Supervised Training of Audio Transformers for Music Representation Learning Pablo Alonso-Jiménez Xavier Serra Dmitry Bogdanov ViT 24 3 0 28 Sep 2023
Weight Averaging Improves Knowledge Distillation under Domain Shift Valeriy Berezovskiy Nikita Morozov MoMe 19 1 0 20 Sep 2023
Interpretability-Aware Vision Transformer Yao Qiang Chengyin Li Prashant Khanduri D. Zhu ViT 80 7 0 14 Sep 2023
3D Transformer based on deformable patch location for differential diagnosis between Alzheimer's disease and Frontotemporal dementia H. Nguyen Michael Clement Boris Mansencal Pierrick Coupé MedIm 26 0 0 06 Sep 2023
Prototype-based Dataset Comparison N. V. Noord 23 6 0 05 Sep 2023
AGS: An Dataset and Taxonomy for Domestic Scene Sound Event Recognition Nan Che Chenrui Liu Fei Yu 25 0 0 30 Aug 2023
Uncovering the Hidden Cost of Model Compression Diganta Misra Muawiz Chaudhary Agam Goyal Bharat Runwal Pin-Yu Chen VLM 24 0 0 29 Aug 2023
PanoSwin: a Pano-style Swin Transformer for Panorama Understanding Zhixin Ling Zhen Xing Xiangdong Zhou Manliang Cao G. Zhou ViT 21 17 0 28 Aug 2023