The Mechanism of Prediction Head in Non-contrastive Self-supervised Learning

12 May 2022

Papers citing "The Mechanism of Prediction Head in Non-contrastive Self-supervised Learning"

34 / 34 papers shown

Title
Projection Head is Secretly an Information Bottleneck Zhuo Ouyang Kaiwen Hu Qi Zhang Yifei Wang Yisen Wang 37 0 0 01 Mar 2025
Direct Coloring for Self-Supervised Enhanced Feature Decoupling S. Mohamadi Gianfranco Doretto Donald Adjeroh 63 0 0 03 Dec 2024
Towards Understanding Why FixMatch Generalizes Better Than Supervised Learning Jingyang Li Jiachun Pan Vincent Y. F. Tan Kim-Chuan Toh Pan Zhou AAML MLT 29 0 0 15 Oct 2024
iQRL -- Implicitly Quantized Representations for Sample-efficient Reinforcement Learning Aidan Scannell Kalle Kujanpää Yi Zhao Mohammadreza Nakhaei Arno Solin J. Pajarinen SSL 24 5 0 04 Jun 2024
What Improves the Generalization of Graph Transformers? A Theoretical Dive into the Self-attention and Positional Encoding Hongkang Li Meng Wang Tengfei Ma Sijia Liu Zaixi Zhang Pin-Yu Chen MLT AI4CE 37 7 0 04 Jun 2024
Investigating the Benefits of Projection Head for Representation Learning Yihao Xue Eric Gan Jiayi Ni Siddharth Joshi Baharan Mirzasoleiman 16 10 0 18 Mar 2024
Rethinking Invariance Regularization in Adversarial Training to Improve Robustness-Accuracy Trade-off Futa Waseda Ching-Chun Chang Isao Echizen AAML 16 0 0 22 Feb 2024
Information Flow in Self-Supervised Learning Zhiyuan Tan Jingqin Yang Weiran Huang Yang Yuan Yifan Zhang SSL 14 13 0 29 Sep 2023
Feature Normalization Prevents Collapse of Non-contrastive Learning Dynamics Han Bao SSL MLT 8 1 0 28 Sep 2023
Diversified Ensemble of Independent Sub-Networks for Robust Self-Supervised Representation Learning Amirhossein Vahidi Lisa Wimmer H. Gündüz Bernd Bischl Eyke Hüllermeier Mina Rezaei OOD UQCV 11 2 0 28 Aug 2023
Towards the Sparseness of Projection Head in Self-Supervised Learning Zeen Song Xingzhe Su Jingyao Wang Wenwen Qiang Changwen Zheng Fuchun Sun 13 3 0 18 Jul 2023
More Synergy, Less Redundancy: Exploiting Joint Mutual Information for Self-Supervised Learning S. Mohamadi Gianfranco Doretto Don Adjeroh SSL 8 4 0 02 Jul 2023
Unraveling Projection Heads in Contrastive Learning: Insights from Expansion and Shrinkage Yu Gui Cong Ma Yiqiao Zhong 6 6 0 06 Jun 2023
Understanding Augmentation-based Self-Supervised Representation Learning via RKHS Approximation and Regression Runtian Zhai Bing Liu Andrej Risteski Zico Kolter Pradeep Ravikumar SSL 20 9 0 01 Jun 2023
Matrix Information Theory for Self-Supervised Learning Yifan Zhang Zhi-Hao Tan Jingqin Yang Weiran Huang Yang Yuan SSL 35 16 0 27 May 2023
Towards a Unified Theoretical Understanding of Non-contrastive Learning via Rank Differential Mechanism Zhijian Zhuo Yifei Wang Jinwen Ma Yisen Wang 32 24 0 04 Mar 2023
InfoNCE Loss Provably Learns Cluster-Preserving Representations Advait Parulekar Liam Collins Karthikeyan Shanmugam Aryan Mokhtari Sanjay Shakkottai SSL 19 12 0 15 Feb 2023
Deciphering the Projection Head: Representation Evaluation Self-supervised Learning Jiajun Ma Tianyang Hu Wenjia Wang 13 5 0 28 Jan 2023
GEDI: GEnerative and DIscriminative Training for Self-Supervised Learning Emanuele Sansone Robin Manhaeve SSL 13 9 0 27 Dec 2022
Understanding Self-Predictive Learning for Reinforcement Learning Yunhao Tang Z. Guo Pierre Harvey Richemond Bernardo Avila-Pires Yash Chandak ... S. Thakoor Will Dabney Bilal Piot Daniele Calandriello Michal Valko SSL 6 28 0 06 Dec 2022
On the Power of Foundation Models Yang Yuan 8 36 0 29 Nov 2022
A Theoretical Study of Inductive Biases in Contrastive Learning Jeff Z. HaoChen Tengyu Ma UQCV SSL 15 31 0 27 Nov 2022
Link Prediction with Non-Contrastive Learning William Shiao Zhichun Guo Tong Zhao Evangelos E. Papalexakis Yozen Liu Neil Shah 11 15 0 25 Nov 2022
Augmentation Invariant Manifold Learning Shulei Wang 19 1 0 01 Nov 2022
BYOL-Explore: Exploration by Bootstrapped Prediction Z. Guo S. Thakoor Miruna Pislar Bernardo Avila-Pires Florent Altché ... Yunhao Tang Michal Valko Rémi Munos M. G. Azar Bilal Piot 11 66 0 16 Jun 2022
Masked Autoencoders Are Scalable Vision Learners Kaiming He Xinlei Chen Saining Xie Yanghao Li Piotr Dollár Ross B. Girshick ViT TPM 258 7,337 0 11 Nov 2021
On Feature Decorrelation in Self-Supervised Learning Tianyu Hua Wenxiao Wang Zihui Xue Sucheng Ren Yue Wang Hang Zhao SSL OOD 107 163 0 02 May 2021
Emerging Properties in Self-Supervised Vision Transformers Mathilde Caron Hugo Touvron Ishan Misra Hervé Jégou Julien Mairal Piotr Bojanowski Armand Joulin 283 5,723 0 29 Apr 2021
Zero-Shot Text-to-Image Generation Aditya A. Ramesh Mikhail Pavlov Gabriel Goh Scott Gray Chelsea Voss Alec Radford Mark Chen Ilya Sutskever VLM 253 4,735 0 24 Feb 2021
Understanding self-supervised Learning Dynamics without Contrastive Pairs Yuandong Tian Xinlei Chen Surya Ganguli SSL 130 258 0 12 Feb 2021
Iterative Graph Self-Distillation Hanlin Zhang Shuai Lin Weiyang Liu Pan Zhou Jian Tang Xiaodan Liang Eric P. Xing SSL 39 33 0 23 Oct 2020
BYOL works even without batch statistics Pierre Harvey Richemond Jean-Bastien Grill Florent Altché Corentin Tallec Florian Strub ... Samuel L. Smith Soham De Razvan Pascanu Bilal Piot Michal Valko SSL 239 114 0 20 Oct 2020
For self-supervised learning, Rationality implies generalization, provably Yamini Bansal Gal Kaplun Boaz Barak OOD SSL 47 22 0 16 Oct 2020
Improved Baselines with Momentum Contrastive Learning Xinlei Chen Haoqi Fan Ross B. Girshick Kaiming He SSL 229 3,029 0 09 Mar 2020