Are All Losses Created Equal: A Neural Collapse Perspective

4 October 2022

Sheng Liu

Qing Qu

Zhihui Zhu

ArXiv PDF HTML

Papers citing "Are All Losses Created Equal: A Neural Collapse Perspective"

50 / 50 papers shown

Title
A Good Start Matters: Enhancing Continual Learning with Data-Driven Weight Initialization Md Yousuf Harun Christopher Kanan AI4CE 48 0 0 09 Mar 2025
Category-free Out-of-Distribution Node Detection with Feature Resonance Shenzhi Yang Junbo Zhao Shouqing Yang Yixuan Li Dingyu Yang Xiaofang Zhang Haobo Wang OODD 41 0 0 22 Feb 2025
MaxSup: Overcoming Representation Collapse in Label Smoothing Yuxuan Zhou Heng Li Zhi-Qi Cheng Xudong Yan Mario Fritz M. Keuper 38 0 0 18 Feb 2025
Multi-layer Radial Basis Function Networks for Out-of-distribution Detection Amol Khanna Chenyi Ling Derek Everett Edward Raff Nathan Inkawhich OODD 23 0 0 05 Jan 2025
Guiding Neural Collapse: Optimising Towards the Nearest Simplex Equiangular Tight Frame Evan Markou Thalaiyasingam Ajanthan Stephen Gould 23 0 0 02 Nov 2024
The Persistence of Neural Collapse Despite Low-Rank Bias: An Analytic Perspective Through Unconstrained Features Connall Garrod Jonathan P. Keating 34 1 0 30 Oct 2024
$$\ell_1$-norm rank-one symmetric matrix factorization has no spurious second-order stationary points$ $\ell_1$ -norm rank-one symmetric matrix factorization has no spurious second-order stationary points Jiewen Guan Anthony Man-Cho So 28 2 0 07 Oct 2024
Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse Arthur Jacot Peter Súkeník Zihan Wang Marco Mondelli 28 1 0 07 Oct 2024
Collapsed Language Models Promote Fairness Jingxuan Xu Wuyang Chen Linyi Li Yao Zhao Yunchao Wei 39 0 0 06 Oct 2024
Exploring Information-Theoretic Metrics Associated with Neural Collapse in Supervised Training Kun Song Zhiquan Tan Bochao Zou Jiansheng Chen Huimin Ma Weiran Huang 30 0 0 25 Sep 2024
The Prevalence of Neural Collapse in Neural Multivariate Regression G. Andriopoulos Zixuan Dong Li Guo Zifan Zhao Keith Ross 30 2 0 06 Sep 2024
Lightweight Uncertainty Quantification with Simplex Semantic Segmentation for Terrain Traversability Judith Dijk Gertjan J. Burghouts Kapil D. Katyal Bryanna Y. Yeh Craig Knuth Ella Fokkinga Tejaswi Kasarla Pascal Mettes 21 0 0 18 Jul 2024
Coding for Intelligence from the Perspective of Category Wenhan Yang Zixuan Hu Lilang Lin Jiaying Liu Ling-Yu Duan AI4CE 30 1 0 01 Jul 2024
Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptation Can Yaras Peng Wang Laura Balzano Qing Qu AI4CE 27 12 0 06 Jun 2024
Unveiling the Dynamics of Information Interplay in Supervised Learning Kun Song Zhiquan Tan Bochao Zou Huimin Ma Weiran Huang 30 1 0 06 Jun 2024
Linguistic Collapse: Neural Collapse in (Large) Language Models Robert Wu V. Papyan 33 11 0 28 May 2024
Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal? Peter Súkeník Marco Mondelli Christoph H. Lampert AI4CE 32 5 0 23 May 2024
Navigate Beyond Shortcuts: Debiased Learning through the Lens of Neural Collapse Yining Wang Junjie Sun Chenyue Wang Mi Zhang Min Yang 21 5 0 09 May 2024
Adapting WavLM for Speech Emotion Recognition Daria Diatlova Anton Udalov Vitalii Shutov Egor Spirin 22 4 0 07 May 2024
Progressive Feedforward Collapse of ResNet Training Sicong Wang Kuo Gai Shihua Zhang AI4CE 25 4 0 02 May 2024
Unifying Low Dimensional Observations in Deep Learning Through the Deep Linear Unconstrained Feature Model Connall Garrod Jonathan P. Keating 22 8 0 09 Apr 2024
Investigating the Benefits of Projection Head for Representation Learning Yihao Xue Eric Gan Jiayi Ni Siddharth Joshi Baharan Mirzasoleiman 26 10 0 18 Mar 2024
Supervised Contrastive Representation Learning: Landscape Analysis with Unconstrained Features Tina Behnia Christos Thrampoulidis SSL 29 0 0 29 Feb 2024
Low-Rank Learning by Design: the Role of Network Architecture and Activation Linearity in Gradient Rank Collapse Bradley T. Baker Ba Pearlmutter Robyn L. Miller Vince D. Calhoun Sergey Plis AI4CE 6 2 0 09 Feb 2024
Pushing Boundaries: Mixup's Influence on Neural Collapse Quinn Fisher Haoming Meng V. Papyan AAML UQCV 31 5 0 09 Feb 2024
Cross Entropy versus Label Smoothing: A Neural Collapse Perspective Li Guo Keith Ross Zifan Zhao G. Andriopoulos Shuyang Ling Yufeng Xu Zixuan Dong UQCV NoLa 17 7 0 06 Feb 2024
Neural Collapse for Cross-entropy Class-Imbalanced Learning with Unconstrained ReLU Feature Model Hien Dang Tho Tran Tan Minh Nguyen Nhat Ho 10 11 0 04 Jan 2024
Convergence Analysis for Learning Orthonormal Deep Linear Neural Networks Zhen Qin Xuwei Tan Zhihui Zhu 19 0 0 24 Nov 2023
Efficient Compression of Overparameterized Deep Models through Low-Dimensional Learning Dynamics Soo Min Kwon Zekai Zhang Dogyoon Song Laura Balzano Qing Qu 30 2 0 08 Nov 2023
Neural Collapse in Multi-label Learning with Pick-all-label Loss Pengyu Li Xiao Li Yutong Wang Qing Qu 10 6 0 24 Oct 2023
A Unified Approach to Domain Incremental Learning with Memory: Theory and Algorithm Haizhou Shi Hao Wang CLL 22 18 0 18 Oct 2023
Towards Demystifying the Generalization Behaviors When Neural Collapse Emerges Peifeng Gao Qianqian Xu Yibo Yang Peisong Wen Huiyang Shao Zhiyong Yang Bernard Ghanem Qingming Huang AAML 10 3 0 12 Oct 2023
Generalized Neural Collapse for a Large Number of Classes Jiachen Jiang Jinxin Zhou Peng Wang Qing Qu Dustin Mixon Chong You Zhihui Zhu AI4CE 24 20 0 09 Oct 2023
Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Data Wanli Hong Shuyang Ling 14 12 0 18 Sep 2023
Neural Collapse in the Intermediate Hidden Layers of Classification Neural Networks Liam Parker Emre Onal Anton Stengel Jake Intrater 15 7 0 05 Aug 2023
Symmetric Neural-Collapse Representations with Supervised Contrastive Loss: The Impact of ReLU and Batching Ganesh Ramachandra Kini V. Vakilian Tina Behnia Jaidev Gill Christos Thrampoulidis 9 1 0 13 Jun 2023
Quantifying the Variability Collapse of Neural Networks Jing-Xue Xu Haoxiong Liu 26 4 0 06 Jun 2023
The Law of Parsimony in Gradient Descent for Learning Deep Linear Networks Can Yaras P. Wang Wei Hu Zhihui Zhu Laura Balzano Qing Qu 20 17 0 01 Jun 2023
Matrix Information Theory for Self-Supervised Learning Yifan Zhang Zhi-Hao Tan Jingqin Yang Weiran Huang Yang Yuan SSL 35 16 0 27 May 2023
Which Features are Learnt by Contrastive Learning? On the Role of Simplicity Bias in Class Collapse and Feature Suppression Yihao Xue S. Joshi Eric Gan Pin-Yu Chen Baharan Mirzasoleiman SSL 23 22 0 25 May 2023
Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Model Peter Súkeník Marco Mondelli Christoph H. Lampert 13 20 0 22 May 2023
A Study of Neural Collapse Phenomenon: Grassmannian Frame, Symmetry and Generalization Peifeng Gao Qianqian Xu Peisong Wen Huiyang Shao Zhiyong Yang Qingming Huang 10 6 0 18 Apr 2023
On the Implicit Geometry of Cross-Entropy Parameterizations for Label-Imbalanced Data Tina Behnia Ganesh Ramachandra Kini V. Vakilian Christos Thrampoulidis 22 17 0 14 Mar 2023
Generalizing and Decoupling Neural Collapse via Hyperspherical Uniformity Gap Weiyang Liu L. Yu Adrian Weller Bernhard Schölkopf 21 15 0 11 Mar 2023
Unsupervised Manifold Linearizing and Clustering Tianjiao Ding Shengbang Tong Kwan Ho Ryan Chan Xili Dai Y. Ma B. Haeffele 16 11 0 04 Jan 2023
Principled and Efficient Transfer Learning of Deep Models via Neural Collapse Xiao Li Sheng Liu Jin-li Zhou Xin Lu C. Fernandez‐Granda Zhihui Zhu Q. Qu AAML 19 18 0 23 Dec 2022
Perturbation Analysis of Neural Collapse Tom Tirer Haoxiang Huang Jonathan Niles-Weed AAML 30 23 0 29 Oct 2022
Memorization-Dilation: Modeling Neural Collapse Under Label Noise Duc Anh Nguyen Ron Levie Julian Lienen Gitta Kutyniok Eyke Hüllermeier 6 1 0 11 Jun 2022
NeuroMixGDP: A Neural Collapse-Inspired Random Mixup for Private Data Release Donghao Li Yang Cao Yuan Yao 28 2 0 14 Feb 2022
An Unconstrained Layer-Peeled Perspective on Neural Collapse Wenlong Ji Yiping Lu Yiliang Zhang Zhun Deng Weijie J. Su 122 83 0 06 Oct 2021