A phase transition between positional and semantic learning in a
solvable model of dot-product attention

A phase transition between positional and semantic learning in a solvable model of dot-product attention

6 February 2024

Florent Krzakala

Lenka Zdeborová

Papers citing "A phase transition between positional and semantic learning in a solvable model of dot-product attention"

6 / 6 papers shown

Title
Physics of Skill Learning Ziming Liu Yizhou Liu Eric J. Michaud Jeff Gore Max Tegmark 41 0 0 21 Jan 2025
Are queries and keys always relevant? A case study on Transformer wave functions Riccardo Rende Luciano Loris Viteritti 24 5 0 29 May 2024
Dissecting the Interplay of Attention Paths in a Statistical Mechanics Theory of Transformers Lorenzo Tiberi Francesca Mignacco Kazuki Irie H. Sompolinsky 42 6 0 24 May 2024
Asymptotic theory of in-context learning by linear attention Yue M. Lu Mary I. Letey Jacob A. Zavatone-Veth Anindita Maiti C. Pehlevan 19 10 0 20 May 2024
The Curious Case of Absolute Position Embeddings Koustuv Sinha Amirhossein Kazemnejad Siva Reddy J. Pineau Dieuwke Hupkes Adina Williams 75 15 0 23 Oct 2022
In-context Learning and Induction Heads Catherine Olsson Nelson Elhage Neel Nanda Nicholas Joseph Nova Dassarma ... Tom B. Brown Jack Clark Jared Kaplan Sam McCandlish C. Olah 240 456 0 24 Sep 2022