Uncovering hidden geometry in Transformers via disentangling position
and context

v1v2 (latest)

Uncovering hidden geometry in Transformers via disentangling position and context

7 October 2023

ArXiv (abs)PDF HTML

Papers citing "Uncovering hidden geometry in Transformers via disentangling position and context"

9 / 9 papers shown

Title
REMA: A Unified Reasoning Manifold Framework for Interpreting Large Language Model Bo Li Guanzhi Deng Ronghao Chen Junrong Yue Shuo Zhang Qinghua Zhao Linqi Song Lijie Wen LRM 85 0 0 26 Sep 2025
Position: Beyond Euclidean -- Foundation Models Should Embrace Non-Euclidean Geometries Neil He Jiahong Liu Buze Zhang N. Bui Ali Maatouk Menglin Yang Irwin King Melanie Weber Rex Ying 187 4 0 11 Apr 2025
Context-aware Biases for Length Extrapolation Ali Veisi Hamidreza Amirzadeh Amir Mansourian 435 1 0 11 Mar 2025
Lines of Thought in Large Language ModelsInternational Conference on Learning Representations (ICLR), 2024 Raphaël Sarfati Toni J. B. Liu Nicolas Boullé Christopher Earls LRM VLM LM&Ro 278 1 0 17 Feb 2025
Out-of-distribution generalization via composition: a lens through induction heads in TransformersProceedings of the National Academy of Sciences of the United States of America (PNAS), 2024 Jiajun Song Zhuoyan Xu Yiqiao Zhong 272 19 0 31 Dec 2024
Reasoning in Large Language Models: A Geometric Perspective Romain Cosentino Sarath Shekkizhar LRM 180 3 0 02 Jul 2024
Transformer Normalisation Layers and the Independence of Semantic Subspaces S. Menary Samuel Kaski Andre Freitas 163 2 0 25 Jun 2024
An Information-Theoretic Analysis of In-Context LearningInternational Conference on Machine Learning (ICML), 2024 Hong Jun Jeon Jason D. Lee Qi Lei Benjamin Van Roy 301 33 0 28 Jan 2024
Characterizing Large Language Model Geometry Helps Solve Toxicity Detection and GenerationInternational Conference on Machine Learning (ICML), 2023 Randall Balestriero Romain Cosentino Sarath Shekkizhar 258 5 0 04 Dec 2023