Training data-efficient image transformers & distillation through attention

23 December 2020

Alexandre Sablayrolles

Hervé Jégou

ViT

ArXiv PDF HTML

Papers citing "Training data-efficient image transformers & distillation through attention"

50 / 1,081 papers shown

Title
CATs++: Boosting Cost Aggregation with Convolutions and Transformers Seokju Cho Sunghwan Hong Seung Wook Kim ViT 19 34 0 14 Feb 2022
How Do Vision Transformers Work? Namuk Park Songkuk Kim ViT 30 465 0 14 Feb 2022
Mixing and Shifting: Exploiting Global and Local Dependencies in Vision MLPs Huangjie Zheng Pengcheng He Weizhu Chen Mingyuan Zhou 22 14 0 14 Feb 2022
Flowformer: Linearizing Transformers with Conservation Flows Haixu Wu Jialong Wu Jiehui Xu Jianmin Wang Mingsheng Long 11 90 0 13 Feb 2022
How to Understand Masked Autoencoders Shuhao Cao Peng-Tao Xu David A. Clifton 23 40 0 08 Feb 2022
LwPosr: Lightweight Efficient Fine-Grained Head Pose Estimation Naina Dhingra 16 16 0 07 Feb 2022
Transformers in Self-Supervised Monocular Depth Estimation with Unknown Camera Intrinsics Arnav Varma Hemang Chawla Bahram Zonooz Elahe Arani ViT MDE 31 49 0 07 Feb 2022
Learning Features with Parameter-Free Layers Dongyoon Han Y. Yoo Beomyoung Kim Byeongho Heo 35 8 0 06 Feb 2022
A Note on "Assessing Generalization of SGD via Disagreement" Andreas Kirsch Y. Gal FedML UQCV 21 15 0 03 Feb 2022
HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection Ke Chen Xingjian Du Bilei Zhu Zejun Ma Taylor Berg-Kirkpatrick Shlomo Dubnov ViT 118 264 0 02 Feb 2022
Query Efficient Decision Based Sparse Attacks Against Black-Box Deep Learning Models Viet Vo Ehsan Abbasnejad D. Ranasinghe AAML 22 14 0 31 Jan 2022
Plug-In Inversion: Model-Agnostic Inversion for Vision with Data Augmentations Amin Ghiasi Hamid Kazemi Steven Reich Chen Zhu Micah Goldblum Tom Goldstein 34 15 0 31 Jan 2022
Aggregating Global Features into Local Vision Transformer Krushi Patel A. Bur Fengju Li Guanghui Wang ViT 25 34 0 30 Jan 2022
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation Junnan Li Dongxu Li Caiming Xiong S. Hoi MLLM BDL VLM CLIP 390 4,125 0 28 Jan 2022
DynaMixer: A Vision MLP Architecture with Dynamic Mixing Ziyu Wang Wenhao Jiang Yiming Zhu Li Yuan Yibing Song Wei Liu 37 43 0 28 Jan 2022
Vision Checklist: Towards Testable Error Analysis of Image Models to Help System Designers Interrogate Model Capabilities Xin Du Bénédicte Legastelois B. Ganesh A. Rajan Hana Chockler Vaishak Belle Stuart Anderson S. Ramamoorthy AAML 19 6 0 27 Jan 2022
Joint Liver and Hepatic Lesion Segmentation in MRI using a Hybrid CNN with Transformer Layers Georg Hille Shubham Agrawal Pavan Tummala C. Wybranski M. Pech A. Surov S. Saalfeld ViT MedIm 11 26 0 26 Jan 2022
One Student Knows All Experts Know: From Sparse to Dense Fuzhao Xue Xiaoxin He Xiaozhe Ren Yuxuan Lou Yang You MoMe MoE 27 20 0 26 Jan 2022
UniFormer: Unifying Convolution and Self-attention for Visual Recognition Kunchang Li Yali Wang Junhao Zhang Peng Gao Guanglu Song Yu Liu Hongsheng Li Yu Qiao ViT 144 361 0 24 Jan 2022
Improving Chest X-Ray Report Generation by Leveraging Warm Starting Aaron Nicolson Jason Dowling Bevan Koopman ViT LM&MA MedIm 22 90 0 24 Jan 2022
VAQF: Fully Automatic Software-Hardware Co-Design Framework for Low-Bit Vision Transformer Mengshu Sun Haoyu Ma Guoliang Kang Yifan Jiang Tianlong Chen Xiaolong Ma Zhangyang Wang Yanzhi Wang ViT 25 45 0 17 Jan 2022
Video Transformers: A Survey Javier Selva A. S. Johansen Sergio Escalera Kamal Nasrollahi T. Moeslund Albert Clapés ViT 22 103 0 16 Jan 2022
UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learning Kunchang Li Yali Wang Peng Gao Guanglu Song Yu Liu Hongsheng Li Yu Qiao ViT 38 238 0 12 Jan 2022
Multiview Transformers for Video Recognition Shen Yan Xuehan Xiong Anurag Arnab Zhichao Lu Mi Zhang Chen Sun Cordelia Schmid ViT 24 211 0 12 Jan 2022
A ConvNet for the 2020s Zhuang Liu Hanzi Mao Chaozheng Wu Christoph Feichtenhofer Trevor Darrell Saining Xie ViT 40 4,967 0 10 Jan 2022
QuadTree Attention for Vision Transformers Shitao Tang Jiahui Zhang Siyu Zhu Ping Tan ViT 161 156 0 08 Jan 2022
Short Range Correlation Transformer for Occluded Person Re-Identification Yunbin Zhao Song-Chun Zhu Dongsheng Wang Zhiwei Liang ViT 13 21 0 04 Jan 2022
PyramidTNT: Improved Transformer-in-Transformer Baselines with Pyramid Architecture Kai Han Jianyuan Guo Yehui Tang Yunhe Wang ViT 26 22 0 04 Jan 2022
Background-aware Classification Activation Map for Weakly Supervised Object Localization Lei Zhu Qi She Qian Chen Xiangxi Meng Mufeng Geng ... Bin Qiu Yunfei You Yibao Zhang Qiushi Ren Yanye Lu WSOL 48 18 0 29 Dec 2021
Pale Transformer: A General Vision Transformer Backbone with Pale-Shaped Attention Sitong Wu Tianyi Wu Hao Hao Tan G. Guo ViT 23 70 0 28 Dec 2021
Vision Transformer for Small-Size Datasets Seung Hoon Lee Seunghyun Lee B. Song ViT 8 222 0 27 Dec 2021
Raw Produce Quality Detection with Shifted Window Self-Attention Oh Joon Kwon Byungsoo Kim Youngduck Choi ViT 22 0 0 24 Dec 2021
ELSA: Enhanced Local Self-Attention for Vision Transformer Jingkai Zhou Pichao Wang Fan Wang Qiong Liu Hao Li Rong Jin ViT 34 37 0 23 Dec 2021
SeMask: Semantically Masked Transformers for Semantic Segmentation Jitesh Jain Anukriti Singh Nikita Orlov Zilong Huang Jiachen Li Steven Walton Humphrey Shi ViT 27 92 0 23 Dec 2021
SLIP: Self-supervision meets Language-Image Pre-training Norman Mu Alexander Kirillov David A. Wagner Saining Xie VLM CLIP 60 476 0 23 Dec 2021
Assessing the Impact of Attention and Self-Attention Mechanisms on the Classification of Skin Lesions Rafael Pedro Arlindo L. Oliveira 17 14 0 23 Dec 2021
RepMLPNet: Hierarchical Vision MLP with Re-parameterized Locality Xiaohan Ding Honghao Chen X. Zhang Jungong Han Guiguang Ding 17 71 0 21 Dec 2021
MPViT: Multi-Path Vision Transformer for Dense Prediction Youngwan Lee Jonghee Kim Jeffrey Willette Sung Ju Hwang ViT 29 244 0 21 Dec 2021
Are Large-scale Datasets Necessary for Self-Supervised Pre-training? Alaaeldin El-Nouby Gautier Izacard Hugo Touvron Ivan Laptev Hervé Jégou Edouard Grave SSL 18 148 0 20 Dec 2021
A Simple Single-Scale Vision Transformer for Object Localization and Instance Segmentation Wuyang Chen Xianzhi Du Fan Yang Lucas Beyer Xiaohua Zhai ... Huizhong Chen Jing Li Xiaodan Song Zhangyang Wang Denny Zhou ViT 21 20 0 17 Dec 2021
Towards End-to-End Image Compression and Analysis with Transformers Yuanchao Bai Xu Yang Xianming Liu Junjun Jiang Yaowei Wang Xiangyang Ji Wen Gao ViT 29 51 0 17 Dec 2021
Masked Feature Prediction for Self-Supervised Visual Pre-Training Chen Wei Haoqi Fan Saining Xie Chaoxia Wu Alan Yuille Christoph Feichtenhofer ViT 74 655 0 16 Dec 2021
Deep Hash Distillation for Image Retrieval Young Kyun Jang Geonmo Gu ByungSoo Ko Isaac Kang N. Cho 19 34 0 16 Dec 2021
FIgLib & SmokeyNet: Dataset and Deep Learning Model for Real-Time Wildland Fire Smoke Detection Anshuman Dewangan Yash Pande Hans-Werner Braun F. Vernon Ismael Pérez I. Altintas G. Cottrell M. H. Nguyen 6 45 0 16 Dec 2021
Towards General and Efficient Active Learning Yichen Xie M. Tomizuka Wei Zhan VLM 32 10 0 15 Dec 2021
Co-training Transformer with Videos and Images Improves Action Recognition Bowen Zhang Jiahui Yu Christopher Fifty Wei Han Andrew M. Dai Ruoming Pang Fei Sha ViT 20 54 0 14 Dec 2021
EMDS-6: Environmental Microorganism Image Dataset Sixth Version for Image Denoising, Segmentation, Feature Extraction, Classification and Detection Methods Evaluation Penghui Zhao Chen Li M. Rahaman Hao Xu Pingli Ma Hechen Yang Hongzan Sun Tao Jiang N. Xu M. Grzegorzek 24 19 0 14 Dec 2021
Short and Long Range Relation Based Spatio-Temporal Transformer for Micro-Expression Recognition Liangfei Zhang Xiaopeng Hong Ognjen Arandjelovic Guoying Zhao ViT 28 47 0 10 Dec 2021
Unified Multimodal Pre-training and Prompt-based Tuning for Vision-Language Understanding and Generation Tianyi Liu Zuxuan Wu Wenhan Xiong Jingjing Chen Yu-Gang Jiang VLM MLLM 32 10 0 10 Dec 2021
Couplformer:Rethinking Vision Transformer with Coupling Attention Map Hai Lan Xihao Wang Xian Wei ViT 26 3 0 10 Dec 2021