An Empirical Study of Training Self-Supervised Vision Transformers

5 April 2021

Papers citing "An Empirical Study of Training Self-Supervised Vision Transformers"

39 / 389 papers shown

Title
PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers Xiaoyi Dong Jianmin Bao Ting Zhang Dongdong Chen Weiming Zhang Lu Yuan Dong Chen Fang Wen Nenghai Yu Baining Guo ViT 39 238 0 24 Nov 2021
Benchmarking Detection Transfer Learning with Vision Transformers Yanghao Li Saining Xie Xinlei Chen Piotr Dollar Kaiming He Ross B. Girshick 14 164 0 22 Nov 2021
Discrete Representations Strengthen Vision Transformer Robustness Chengzhi Mao Lu Jiang Mostafa Dehghani Carl Vondrick Rahul Sukthankar Irfan Essa ViT 25 43 0 20 Nov 2021
SimMIM: A Simple Framework for Masked Image Modeling Zhenda Xie Zheng-Wei Zhang Yue Cao Yutong Lin Jianmin Bao Zhuliang Yao Qi Dai Han Hu 37 1,309 0 18 Nov 2021
iBOT: Image BERT Pre-Training with Online Tokenizer Jinghao Zhou Chen Wei Huiyu Wang Wei Shen Cihang Xie Alan Yuille Tao Kong 19 709 0 15 Nov 2021
Masked Autoencoders Are Scalable Vision Learners Kaiming He Xinlei Chen Saining Xie Yanghao Li Piotr Dollár Ross B. Girshick ViT TPM 305 7,434 0 11 Nov 2021
A Survey of Visual Transformers Yang Liu Yao Zhang Yixin Wang Feng Hou Jin Yuan Jiang Tian Yang Zhang Zhongchao Shi Jianping Fan Zhiqiang He 3DGS ViT 69 330 0 11 Nov 2021
Probabilistic Contrastive Learning for Domain Adaptation Junjie Li Yixin Zhang Zilei Wang Saihui Hou Keyu Tu Man Zhang 28 14 0 11 Nov 2021
SSAST: Self-Supervised Audio Spectrogram Transformer Yuan Gong Cheng-I Jeff Lai Yu-An Chung James R. Glass ViT 30 268 0 19 Oct 2021
Self-Supervised Learning by Estimating Twin Class Distributions Feng Wang Tao Kong Rufeng Zhang Huaping Liu Hang Li SSL 50 16 0 14 Oct 2021
The Impact of Spatiotemporal Augmentations on Self-Supervised Audiovisual Representation Learning Haider Al-Tahan Y. Mohsenzadeh SSL AI4TS 27 0 0 13 Oct 2021
Dynamic Inference with Neural Interpreters Nasim Rahaman Muhammad Waleed Gondal S. Joshi Peter V. Gehler Yoshua Bengio Francesco Locatello Bernhard Schölkopf 31 31 0 12 Oct 2021
Revitalizing CNN Attentions via Transformers in Self-Supervised Visual Representation Learning Chongjian Ge Youwei Liang Yibing Song Jianbo Jiao Jue Wang Ping Luo ViT 16 36 0 11 Oct 2021
Homography augumented momentum constrastive learning for SAR image retrieval Seonho Park M. Rysz Kathleen M. Dipple P. Pardalos 23 1 0 21 Sep 2021
Self supervised learning improves dMMR/MSI detection from histology slides across multiple cancers C. Saillard Olivier Dehaene Tanguy Marchand O. Moindrot A. Kamoun B. Schmauch S. Jégou 30 39 0 13 Sep 2021
Do Vision Transformers See Like Convolutional Neural Networks? M. Raghu Thomas Unterthiner Simon Kornblith Chiyuan Zhang Alexey Dosovitskiy ViT 41 922 0 19 Aug 2021
On the Efficacy of Small Self-Supervised Contrastive Models without Distillation Signals Haizhou Shi Youcai Zhang Siliang Tang Wenjie Zhu Yaqian Li Yandong Guo Yueting Zhuang SyDa 21 14 0 30 Jul 2021
Focal Self-attention for Local-Global Interactions in Vision Transformers Jianwei Yang Chunyuan Li Pengchuan Zhang Xiyang Dai Bin Xiao Lu Yuan Jianfeng Gao ViT 42 428 0 01 Jul 2021
Efficient Self-supervised Vision Transformers for Representation Learning Chunyuan Li Jianwei Yang Pengchuan Zhang Mei Gao Bin Xiao Xiyang Dai Lu Yuan Jianfeng Gao ViT 30 209 0 17 Jun 2021
BEiT: BERT Pre-Training of Image Transformers Hangbo Bao Li Dong Songhao Piao Furu Wei ViT 30 2,744 0 15 Jun 2021
D2C: Diffusion-Denoising Models for Few-shot Conditional Generation Abhishek Sinha Jiaming Song Chenlin Meng Stefano Ermon VLM DiffM 19 118 0 12 Jun 2021
Learning to See by Looking at Noise Manel Baradad Jonas Wulff Tongzhou Wang Phillip Isola Antonio Torralba 18 89 0 10 Jun 2021
MST: Masked Self-Supervised Transformer for Visual Representation Zhaowen Li Zhiyang Chen Fan Yang Wei Li Yousong Zhu ... Rui Deng Liwei Wu Rui Zhao Ming Tang Jinqiao Wang ViT 30 161 0 10 Jun 2021
ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias Yufei Xu Qiming Zhang Jing Zhang Dacheng Tao ViT 48 329 0 07 Jun 2021
Signal Transformer: Complex-valued Attention and Meta-Learning for Signal Recognition Yihong Dong Ying Peng Muqiao Yang Songtao Lu Qingjiang Shi 38 9 0 05 Jun 2021
Analogous to Evolutionary Algorithm: Designing a Unified Sequence Model Jiangning Zhang Chao Xu Jian Li Wenzhou Chen Yabiao Wang Ying Tai Shuo Chen Chengjie Wang Feiyue Huang Yong Liu 27 22 0 31 May 2021
Divide and Contrast: Self-supervised Learning from Uncurated Data Yonglong Tian Olivier J. Hénaff Aaron van den Oord SSL 51 96 0 17 May 2021
Contrastive Attraction and Contrastive Repulsion for Representation Learning Huangjie Zheng Xu Chen Jiangchao Yao Hongxia Yang Chunyuan Li Ya-Qin Zhang Hao Zhang Ivor Tsang Jingren Zhou Mingyuan Zhou SSL 34 12 0 08 May 2021
Salient Objects in Clutter Deng-Ping Fan Jing Zhang Gang Xu Mingg-Ming Cheng Ling Shao 29 42 0 07 May 2021
Beyond Self-attention: External Attention using Two Linear Layers for Visual Tasks Meng-Hao Guo Zheng-Ning Liu Tai-Jiang Mu Shimin Hu 20 473 0 05 May 2021
Emerging Properties in Self-Supervised Vision Transformers Mathilde Caron Hugo Touvron Ishan Misra Hervé Jégou Julien Mairal Piotr Bojanowski Armand Joulin 305 5,773 0 29 Apr 2021
ImageNet-21K Pretraining for the Masses T. Ridnik Emanuel Ben-Baruch Asaf Noy Lihi Zelnik-Manor SSeg VLM CLIP 173 686 0 22 Apr 2021
Vision Transformer using Low-level Chest X-ray Feature Corpus for COVID-19 Diagnosis and Severity Quantification Sangjoon Park Gwanghyun Kim Y. Oh J. Seo Sang Min Lee Jin Hwan Kim Sungjun Moon Jae-Kwang Lim Jong Chul Ye ViT MedIm 40 96 0 15 Apr 2021
SiT: Self-supervised vIsion Transformer Sara Atito Ali Ahmed Muhammad Awais J. Kittler ViT 31 139 0 08 Apr 2021
Characterizing and Improving the Robustness of Self-Supervised Learning through Background Augmentations Chaitanya K. Ryali D. Schwab Ari S. Morcos SSL 24 9 0 23 Mar 2021
Cluster Contrast for Unsupervised Person Re-Identification Zuozhuo Dai Guangyuan Wang Weihao Yuan Xiaoli Liu Siyu Zhu P. Tan 21 212 0 22 Mar 2021
Transformers in Vision: A Survey Salman Khan Muzammal Naseer Munawar Hayat Syed Waqas Zamir F. Khan M. Shah ViT 227 2,428 0 04 Jan 2021
Exploring Simple Siamese Representation Learning Xinlei Chen Kaiming He SSL 14 3,949 0 20 Nov 2020
Improved Baselines with Momentum Contrastive Learning Xinlei Chen Haoqi Fan Ross B. Girshick Kaiming He SSL 264 3,369 0 09 Mar 2020