PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers

24 November 2021

Jianmin Bao

Lu Yuan

Papers citing "PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers"

50 / 189 papers shown

Title
Masked Event Modeling: Self-Supervised Pretraining for Event Cameras Simone Klenk David Bonello Lukas Koestler Nikita Araslanov Daniel Cremers 17 22 0 20 Dec 2022
What do Vision Transformers Learn? A Visual Exploration Amin Ghiasi Hamid Kazemi Eitan Borgnia Steven Reich Manli Shu Micah Goldblum A. Wilson Tom Goldstein ViT 11 60 0 13 Dec 2022
FastMIM: Expediting Masked Image Modeling Pre-training for Vision Jianyuan Guo Kai Han Han Wu Yehui Tang Yunhe Wang Chang Xu 10 8 0 13 Dec 2022
CLIP Itself is a Strong Fine-tuner: Achieving 85.7% and 88.0% Top-1 Accuracy with ViT-B and ViT-L on ImageNet Xiaoyi Dong Jianmin Bao Ting Zhang Dongdong Chen Shuyang Gu Weiming Zhang Lu Yuan Dong Chen Fang Wen Nenghai Yu CLIP 14 35 0 12 Dec 2022
Masked Video Distillation: Rethinking Masked Feature Modeling for Self-supervised Video Representation Learning Rui Wang Dongdong Chen Zuxuan Wu Yinpeng Chen Xiyang Dai Mengchen Liu Lu Yuan Yu-Gang Jiang VGen 19 86 0 08 Dec 2022
Rethinking the Objectives of Vector-Quantized Tokenizers for Image Synthesis Yuchao Gu Xintao Wang Yixiao Ge Ying Shan Xiaohu Qie Mike Zheng Shou DiffM 13 20 0 06 Dec 2022
MIC: Masked Image Consistency for Context-Enhanced Domain Adaptation Lukas Hoyer Dengxin Dai Haoran Wang Luc Van Gool 25 217 0 02 Dec 2022
Self-Supervised Learning based on Heat Equation Yinpeng Chen Xiyang Dai Dongdong Chen Mengchen Liu Lu Yuan Zicheng Liu Youzuo Lin 16 4 0 23 Nov 2022
Fast-iTPN: Integrally Pre-Trained Transformer Pyramid Network with Token Migration Yunjie Tian Lingxi Xie Jihao Qiu Jianbin Jiao Yaowei Wang Qi Tian Qixiang Ye ViT 19 6 0 23 Nov 2022
Contrastive Masked Autoencoders for Self-Supervised Video Hashing Yuting Wang Jinpeng Wang B. Chen Ziyun Zeng Shutao Xia 13 20 0 21 Nov 2022
Peeling the Onion: Hierarchical Reduction of Data Redundancy for Efficient Vision Transformer Training Zhenglun Kong Haoyu Ma Geng Yuan Mengshu Sun Yanyue Xie ... Tianlong Chen Xiaolong Ma Xiaohui Xie Zhangyang Wang Yanzhi Wang ViT 15 22 0 19 Nov 2022
CAE v2: Context Autoencoder with CLIP Target Xinyu Zhang Jiahui Chen Junkun Yuan Qiang Chen Jian Wang ... Jimin Pi Kun Yao Junyu Han Errui Ding Jingdong Wang VLM CLIP 34 24 0 17 Nov 2022
MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesis Tianhong Li Huiwen Chang Shlok Kumar Mishra Han Zhang Dina Katabi Dilip Krishnan 16 149 0 16 Nov 2022
Stare at What You See: Masked Image Modeling without Reconstruction Hongwei Xue Peng Gao Hongyang Li Yu Qiao Hao Sun Houqiang Li Jiebo Luo 22 31 0 16 Nov 2022
Artificial intelligence approaches for materials-by-design of energetic materials: state-of-the-art, challenges, and future directions Joseph B. Choi Phong C. H. Nguyen O. Sen H. Udaykumar Stephen Seung-Yeob Baek PINN AI4CE 14 11 0 15 Nov 2022
EVA: Exploring the Limits of Masked Visual Representation Learning at Scale Yuxin Fang Wen Wang Binhui Xie Quan-Sen Sun Ledell Yu Wu Xinggang Wang Tiejun Huang Xinlong Wang Yue Cao VLM CLIP 46 671 0 14 Nov 2022
Masked Contrastive Representation Learning Yuan Yao Nandakishor Desai M. Palaniswami SSL 14 8 0 11 Nov 2022
Towards Sustainable Self-supervised Learning Shanghua Gao Pan Zhou Mingg-Ming Cheng Shuicheng Yan CLL 25 7 0 20 Oct 2022
A Unified View of Masked Image Modeling Zhiliang Peng Li Dong Hangbo Bao QiXiang Ye Furu Wei VLM 52 35 0 19 Oct 2022
Exploring Long-Sequence Masked Autoencoders Ronghang Hu Shoubhik Debnath Saining Xie Xinlei Chen 6 17 0 13 Oct 2022
Self-supervised Video Representation Learning with Motion-Aware Masked Autoencoders Haosen Yang Deng Huang Bin Wen Jiannan Wu H. Yao Yi-Xin Jiang Xiatian Zhu Zehuan Yuan 24 19 0 09 Oct 2022
Progressive Text-to-Image Generation Zhengcong Fei Mingyuan Fan Li Zhu Junshi Huang 65 4 0 05 Oct 2022
Exploring The Role of Mean Teachers in Self-supervised Masked Auto-Encoders Youngwan Lee Jeffrey Willette Jonghee Kim Juho Lee Sung Ju Hwang 18 16 0 05 Oct 2022
Attention Distillation: self-supervised vision transformer students need more guidance Kai Wang Fei Yang Joost van de Weijer ViT 17 14 0 03 Oct 2022
OmniVL:One Foundation Model for Image-Language and Video-Language Tasks Junke Wang Dongdong Chen Zuxuan Wu Chong Luo Luowei Zhou Yucheng Zhao Yujia Xie Ce Liu Yu-Gang Jiang Lu Yuan MLLM VLM 27 148 0 15 Sep 2022
MimCo: Masked Image Modeling Pre-training with Contrastive Teacher Qiang-feng Zhou Chaohui Yu Haowen Luo Zhibin Wang Hao Li VLM 54 20 0 07 Sep 2022
MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretraining Xiaoyi Dong Jianmin Bao Yinglin Zheng Ting Zhang Dongdong Chen ... Weiming Zhang Lu Yuan Dong Chen Fang Wen Nenghai Yu CLIP VLM 32 156 0 25 Aug 2022
Video Mobile-Former: Video Recognition with Efficient Global Spatial-temporal Modeling Rui Wang Zuxuan Wu Dongdong Chen Yinpeng Chen Xiyang Dai Mengchen Liu Luowei Zhou Lu Yuan Yu-Gang Jiang ViT 30 4 0 25 Aug 2022
VLMAE: Vision-Language Masked Autoencoder Su He Taian Guo Tao Dai Ruizhi Qiao Chen Wu Xiujun Shu Bohan Ren VLM 11 11 0 19 Aug 2022
Towards Label-efficient Automatic Diagnosis and Analysis: A Comprehensive Survey of Advanced Deep Learning-based Weakly-supervised, Semi-supervised and Self-supervised Techniques in Histopathological Image Analysis Linhao Qu Siyu Liu Xiaoyu Liu Manning Wang Zhijian Song 15 56 0 18 Aug 2022
BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers Zhiliang Peng Li Dong Hangbo Bao QiXiang Ye Furu Wei 16 301 0 12 Aug 2022
MILAN: Masked Image Pretraining on Language Assisted Representation Zejiang Hou Fei Sun Yen-kuang Chen Yuan Xie S. Kung ViT 13 66 0 11 Aug 2022
Understanding Masked Image Modeling via Learning Occlusion Invariant Feature Xiangwen Kong Xiangyu Zhang SSL 6 51 0 08 Aug 2022
Augmenting Vision Language Pretraining by Learning Codebook with Visual Semantics Xiaoyuan Guo Jiali Duan C.-C. Jay Kuo J. Gichoya Imon Banerjee VLM 8 1 0 31 Jul 2022
SdAE: Self-distillated Masked Autoencoder Yabo Chen Yuchen Liu Dongsheng Jiang Xiaopeng Zhang Wenrui Dai H. Xiong Qi Tian ViT 13 70 0 31 Jul 2022
A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyond Chaoning Zhang Chenshuang Zhang Junha Song John Seon Keun Yi Kang Zhang In So Kweon SSL 39 70 0 30 Jul 2022
Contrastive Masked Autoencoders are Stronger Vision Learners Zhicheng Huang Xiaojie Jin Cheng Lu Qibin Hou Mingg-Ming Cheng Dongmei Fu Xiaohui Shen Jiashi Feng 26 146 0 27 Jul 2022
FashionViL: Fashion-Focused Vision-and-Language Representation Learning Xiaoping Han Licheng Yu Xiatian Zhu Li Zhang Yi-Zhe Song Tao Xiang AI4TS 16 49 0 17 Jul 2022
Bootstrapped Masked Autoencoders for Vision BERT Pretraining Xiaoyi Dong Jianmin Bao Ting Zhang Dongdong Chen Weiming Zhang Lu Yuan Dong Chen Fang Wen Nenghai Yu 17 74 0 14 Jul 2022
Dissecting Self-Supervised Learning Methods for Surgical Computer Vision Sanat Ramesh V. Srivastav Deepak Alapatt Tong Yu Aditya Murali ... Saurav Sharma A. Fleurentin Georgios Exarchakis Alexandros Karargyris N. Padoy 16 42 0 01 Jul 2022
SemMAE: Semantic-Guided Masking for Learning Masked Autoencoders Gang Li Heliang Zheng Daqing Liu Chaoyue Wang Bing-Huang Su Changwen Zheng 19 124 0 21 Jun 2022
EATFormer: Improving Vision Transformer Inspired by Evolutionary Algorithm Jiangning Zhang Xiangtai Li Yabiao Wang Chengjie Wang Yibo Yang Yong Liu Dacheng Tao ViT 28 32 0 19 Jun 2022
Masked Frequency Modeling for Self-Supervised Visual Pre-Training Jiahao Xie Wei Li Xiaohang Zhan Ziwei Liu Yew-Soon Ong Chen Change Loy 11 69 0 15 Jun 2022
Extreme Masking for Learning Instance and Distributed Visual Representations Zhirong Wu Zihang Lai Xiao Sun Stephen Lin 24 22 0 09 Jun 2022
Spatial Entropy as an Inductive Bias for Vision Transformers E. Peruzzo E. Sangineto Yahui Liu Marco De Nadai Wei Bi Bruno Lepri N. Sebe ViT MDE 17 1 0 09 Jun 2022
Towards Understanding Why Mask-Reconstruction Pretraining Helps in Downstream Tasks Jia-Yu Pan Pan Zhou Shuicheng Yan SSL 6 14 0 08 Jun 2022
Siamese Image Modeling for Self-Supervised Vision Representation Learning Chenxin Tao Xizhou Zhu Weijie Su Gao Huang Bin Li Jie Zhou Yu Qiao Xiaogang Wang Jifeng Dai SSL 24 94 0 02 Jun 2022
Transforming medical imaging with Transformers? A comparative review of key properties, current progresses, and future perspectives Jun Li Junyu Chen Yucheng Tang Ce Wang Bennett A. Landman S. K. Zhou ViT OOD MedIm 11 18 0 02 Jun 2022
Efficient Self-supervised Vision Pretraining with Local Masked Reconstruction Jun Chen Ming Hu Boyang Albert Li Mohamed Elhoseiny 25 36 0 01 Jun 2022
SupMAE: Supervised Masked Autoencoders Are Efficient Vision Learners Feng Liang Yangguang Li Diana Marculescu SSL TPM ViT 40 22 0 28 May 2022