The polysemantic nature of synthetic neurons in artificial intelligence language models is currently understood as the result of a necessary superposition of distributed features within the latent space. We propose an alternative approach, geometrically defining a neuron in layer n as a categorical vector space with a non-orthogonal basis, composed of categorical sub-dimensions extracted from preceding neurons in layer n-1. This categorical vector space is structured by the activation space of each neuron and enables, via an intra-neuronal attention process, the identification and utilization of a critical categorical zone for the efficiency of the language model - more homogeneous and located at the intersection of these different categorical sub-dimensions.
View on arXiv@article{pichat2025_2505.07831,
title={ Polysemy of Synthetic Neurons Towards a New Type of Explanatory Categorical Vector Spaces },
author={ Michael Pichat and William Pogrund and Paloma Pichat and Judicael Poumay and Armanouche Gasparian and Samuel Demarchi and Martin Corbet and Alois Georgeon and Michael Veillet-Guillem },
journal={arXiv preprint arXiv:2505.07831},
year={ 2025 }
}