Linguistic Collapse: Neural Collapse in (Large) Language Models

28 May 2024

Papers citing "Linguistic Collapse: Neural Collapse in (Large) Language Models"

21 / 21 papers shown

Title
Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representations Yize Zhao Tina Behnia V. Vakilian Christos Thrampoulidis 53 7 0 20 Feb 2025
Parameter Symmetry Breaking and Restoration Determines the Hierarchical Learning in AI Systems Liu Ziyin Yizhou Xu T. Poggio Isaac Chuang 48 4 0 07 Feb 2025
The Persistence of Neural Collapse Despite Low-Rank Bias: An Analytic Perspective Through Unconstrained Features Connall Garrod Jonathan P. Keating 34 1 0 30 Oct 2024
Mitigating Gender Bias in Code Large Language Models via Model Editing Z. Qin Haochuan Wang Zecheng Wang Deyuan Liu Cunhang Fan Zhao Lv Zhiying Tu Dianhui Chu Dianbo Sui KELM 18 0 0 10 Oct 2024
Control-oriented Clustering of Visual Latent Representation Han Qi Haocheng Yin Heng Yang SSL 48 2 0 07 Oct 2024
Collapsed Language Models Promote Fairness Jingxuan Xu Wuyang Chen Linyi Li Yao Zhao Yunchao Wei 39 0 0 06 Oct 2024
A Law of Next-Token Prediction in Large Language Models Hangfeng He Weijie J. Su 19 5 0 24 Aug 2024
Words in Motion: Extracting Interpretable Control Vectors for Motion Transformers Omer Sahin Tas Royden Wagner 39 1 0 17 Jun 2024
The Impact of Geometric Complexity on Neural Collapse in Transfer Learning Michael Munn Benoit Dherin Javier Gonzalvo AAML 27 1 0 24 May 2024
Neural Rank Collapse: Weight Decay and Small Within-Class Variability Yield Low-Rank Bias Emanuele Zangrando Piero Deidda Simone Brugiapaglia Nicola Guglielmi Francesco Tudisco 14 8 0 06 Feb 2024
On the Role of Neural Collapse in Meta Learning Models for Few-shot Learning Saaketh Medepalli Naren Doraiswamy 18 1 0 30 Sep 2023
Inducing Neural Collapse to a Fixed Hierarchy-Aware Frame for Reducing Mistake Severity Tong Liang Jim Davis 25 10 0 10 Mar 2023
Understanding Imbalanced Semantic Segmentation Through Neural Collapse Zhisheng Zhong Jiequan Cui Yibo Yang Xiaoyang Wu Xiaojuan Qi X. Zhang Jiaya Jia 119 44 0 03 Jan 2023
Perturbation Analysis of Neural Collapse Tom Tirer Haoxiang Huang Jonathan Niles-Weed AAML 24 23 0 29 Oct 2022
Toy Models of Superposition Nelson Elhage Tristan Hume Catherine Olsson Nicholas Schiefer T. Henighan ... Sam McCandlish Jared Kaplan Dario Amodei Martin Wattenberg C. Olah AAML MILM 117 314 0 21 Sep 2022
Linking Neural Collapse and L2 Normalization with Improved Out-of-Distribution Detection in Deep Neural Networks J. Haas William Yolland B. Rabus OODD 41 14 0 17 Sep 2022
NeuroMixGDP: A Neural Collapse-Inspired Random Mixup for Private Data Release Donghao Li Yang Cao Yuan Yao 22 2 0 14 Feb 2022
Nearest Class-Center Simplification through Intermediate Layers Ido Ben-Shaul S. Dekel 27 26 0 21 Jan 2022
Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Training Cong Fang Hangfeng He Qi Long Weijie J. Su FAtt 112 162 0 29 Jan 2021
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 220 3,054 0 23 Jan 2020
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 294 6,927 0 20 Apr 2018