On the Stepwise Nature of Self-Supervised Learning

27 March 2023

Papers citing "On the Stepwise Nature of Self-Supervised Learning"

31 / 31 papers shown

Title
An Empirically Grounded Identifiability Theory Will Accelerate Self-Supervised Learning Research Patrik Reizinger Randall Balestriero David Klindt Wieland Brendel 40 0 0 17 Apr 2025
Position: Solve Layerwise Linear Models First to Understand Neural Dynamical Phenomena (Neural Collapse, Emergence, Lazy/Rich Regime, and Grokking) Yoonsoo Nam Seok Hyeong Lee Clementine Domine Yea Chan Park Charles London Wonyl Choi Niclas Goring Seungjai Lee AI4CE 38 0 0 28 Feb 2025
Parameter Symmetry Breaking and Restoration Determines the Hierarchical Learning in AI Systems Liu Ziyin Yizhou Xu T. Poggio Isaac Chuang 50 4 0 07 Feb 2025
It's Not Just a Phase: On Investigating Phase Transitions in Deep Learning-based Side-channel Analysis Sengim Karayalçin Marina Krček Stjepan Picek AAML 70 0 0 01 Feb 2025
Infinite Width Limits of Self Supervised Neural Networks Maximilian Fleissner Gautham Govind Anil D. Ghoshdastidar SSL 142 0 0 17 Nov 2024
Cross-Entropy Is All You Need To Invert the Data Generating Process Patrik Reizinger Alice Bizeul Attila Juhos Julia E. Vogt Randall Balestriero Wieland Brendel David Klindt SSL OOD BDL DRL 102 3 0 29 Oct 2024
The Optimization Landscape of SGD Across the Feature Learning Strength Alexander B. Atanasov Alexandru Meterez James B. Simon C. Pehlevan 43 2 0 06 Oct 2024
Towards Automatic Assessment of Self-Supervised Speech Models using Rank Zakaria Aldeneh Vimal Thilak Takuya Higuchi B. Theobald Tatiana Likhomanenko SSL 75 0 0 16 Sep 2024
How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networks Etai Littwin Omid Saremi Madhu Advani Vimal Thilak Preetum Nakkiran Chen Huang Joshua Susskind 37 3 0 03 Jul 2024
InfoNCE: Identifying the Gap Between Theory and Practice E. Rusak Patrik Reizinger Attila Juhos Oliver Bringmann Roland S. Zimmermann Wieland Brendel 33 5 0 28 Jun 2024
Training Dynamics of Nonlinear Contrastive Learning Model in the High Dimensional Limit Lineghuan Meng Chuang Wang 23 1 0 11 Jun 2024
Phase Transitions in the Output Distribution of Large Language Models Julian Arnold Flemming Holtorf Frank Schafer Niels Lörch 41 1 0 27 May 2024
Mechanistic Interpretability for AI Safety -- A Review Leonard Bereska E. Gavves AI4CE 40 111 0 22 Apr 2024
Auxiliary task demands mask the capabilities of smaller language models Jennifer Hu Michael C. Frank ELM 34 25 0 03 Apr 2024
When can we Approximate Wide Contrastive Models with Neural Tangent Kernels and Principal Component Analysis? Gautham Govind Anil P. Esser D. Ghoshdastidar 29 1 0 13 Mar 2024
Compression of Structured Data with Autoencoders: Provable Benefit of Nonlinearities and Depth Kevin Kögler A. Shevchenko Hamed Hassani Marco Mondelli MLT 25 0 0 07 Feb 2024
FroSSL: Frobenius Norm Minimization for Efficient Multiview Self-Supervised Learning Oscar Skean A. Dhakal Nathan Jacobs Luis Gonzalo Sánchez Giraldo 29 0 0 04 Oct 2023
Representation Learning Dynamics of Self-Supervised Models P. Esser Satyaki Mukherjee D. Ghoshdastidar SSL 19 2 0 05 Sep 2023
Transformers learn through gradual rank increase Enric Boix-Adserà Etai Littwin Emmanuel Abbe Samy Bengio J. Susskind 38 33 0 12 Jun 2023
The Quantization Model of Neural Scaling Eric J. Michaud Ziming Liu Uzay Girit Max Tegmark MILM 27 77 0 23 Mar 2023
The SSL Interplay: Augmentations, Inductive Bias, and Generalization Vivien A. Cabannes B. Kiani Randall Balestriero Yann LeCun A. Bietti SSL 11 31 0 06 Feb 2023
Neural networks learn to magnify areas near decision boundaries Jacob A. Zavatone-Veth Sheng Yang Julian Rubinfien C. Pehlevan MLT AI4CE 20 6 0 26 Jan 2023
Implicit variance regularization in non-contrastive SSL Manu Srinath Halvagal Axel Laborieux Friedemann Zenke 34 8 0 09 Dec 2022
RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rank Q. Garrido Randall Balestriero Laurent Najman Yann LeCun SSL 46 72 0 05 Oct 2022
What shapes the loss landscape of self-supervised learning? Liu Ziyin Ekdeep Singh Lubana Masakuni Ueda Hidenori Tanaka 48 20 0 02 Oct 2022
Joint Embedding Self-Supervised Learning in the Kernel Regime B. Kiani Randall Balestriero Yubei Chen S. Lloyd Yann LeCun SSL 41 13 0 29 Sep 2022
Masked Autoencoders Are Scalable Vision Learners Kaiming He Xinlei Chen Saining Xie Yanghao Li Piotr Dollár Ross B. Girshick ViT TPM 305 7,434 0 11 Nov 2021
The Eigenlearning Framework: A Conservation Law Perspective on Kernel Regression and Wide Neural Networks James B. Simon Madeline Dickens Dhruva Karkada M. DeWeese 40 27 0 08 Oct 2021
Emerging Properties in Self-Supervised Vision Transformers Mathilde Caron Hugo Touvron Ishan Misra Hervé Jégou Julien Mairal Piotr Bojanowski Armand Joulin 308 5,773 0 29 Apr 2021
Understanding self-supervised Learning Dynamics without Contrastive Pairs Yuandong Tian Xinlei Chen Surya Ganguli SSL 138 279 0 12 Feb 2021
On the surprising similarities between supervised and self-supervised models Robert Geirhos Kantharaju Narayanappa Benjamin Mitzkus Matthias Bethge Felix Wichmann Wieland Brendel OOD SSL DRL 69 46 0 16 Oct 2020