Looped Transformers are Better at Learning Learning Algorithms

21 November 2023

Liu Yang

Kangwook Lee

Robert D. Nowak

Dimitris Papailiopoulos

ArXiv PDF HTML

Papers citing "Looped Transformers are Better at Learning Learning Algorithms"

20 / 20 papers shown

Title
Intra-Layer Recurrence in Transformers for Language Modeling Anthony Nguyen Wenjun Lin 15 0 0 03 May 2025
A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformers William Merrill Ashish Sabharwal 53 4 0 05 Mar 2025
In-Context Learning with Hypothesis-Class Guidance Ziqian Lin Shubham Kumar Bharti Kangwook Lee 69 0 0 27 Feb 2025
Exact Learning of Permutations for Nonzero Binary Inputs with Logarithmic Training Size and Quadratic Ensemble Complexity George Giapitzakis Artur Back de Luca K. Fountoulakis 41 0 0 24 Feb 2025
Reasoning with Latent Thoughts: On the Power of Looped Transformers Nikunj Saunshi Nishanth Dikkala Zhiyuan Li Sanjiv Kumar Sashank J. Reddi OffRL LRM AI4CE 45 9 0 24 Feb 2025
Enhancing Auto-regressive Chain-of-Thought through Loop-Aligned Reasoning Qifan Yu Zhenyu He Sijie Li Xun Zhou Jun Zhang Jingjing Xu Di He OffRL LRM 86 4 0 12 Feb 2025
On the Role of Depth and Looping for In-Context Learning with Task Diversity Khashayar Gatmiry Nikunj Saunshi Sashank J. Reddi Stefanie Jegelka Sanjiv Kumar 22 2 0 29 Oct 2024
Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA Sangmin Bae Adam Fisch Hrayr Harutyunyan Ziwei Ji Seungyeon Kim Tal Schuster KELM 68 5 0 28 Oct 2024
Advancing the Understanding of Fixed Point Iterations in Deep Neural Networks: A Detailed Analytical Study Yekun Ke Xiaoyu Li Yingyu Liang Zhenmei Shi Zhao-quan Song 58 3 0 15 Oct 2024
Bypassing the Exponential Dependency: Looped Transformers Efficiently Learn In-context by Multi-step Gradient Descent Bo Chen Xiaoyu Li Yingyu Liang Zhenmei Shi Zhao-quan Song 83 19 0 15 Oct 2024
Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning? Khashayar Gatmiry Nikunj Saunshi Sashank J. Reddi Stefanie Jegelka Sanjiv Kumar 67 17 0 10 Oct 2024
Everything Everywhere All at Once: LLMs can In-Context Learn Multiple Tasks in Superposition Zheyang Xiong Ziyang Cai John Cooper Albert Ge Vasilis Papageorgiou ... Saurabh Agarwal Grigorios G Chrysos Samet Oymak Kangwook Lee Dimitris Papailiopoulos LRM 22 1 0 08 Oct 2024
On Expressive Power of Looped Transformers: Theoretical Analysis and Enhancement via Timestep Encoding Kevin Xu Issei Sato 37 3 0 02 Oct 2024
Transformers Handle Endogeneity in In-Context Linear Regression Haodong Liang Krishnakumar Balasubramanian Lifeng Lai 32 1 0 02 Oct 2024
Positional Attention: Expressivity and Learnability of Algorithmic Computation Artur Back de Luca George Giapitzakis Shenghao Yang Petar Veličković K. Fountoulakis 37 0 0 02 Oct 2024
On the Inductive Bias of Stacking Towards Improving Reasoning Nikunj Saunshi Stefani Karp Shankar Krishnan Sobhan Miryoosefi Sashank J. Reddi Sanjiv Kumar LRM AI4CE 29 4 0 27 Sep 2024
Revisiting the Graph Reasoning Ability of Large Language Models: Case Studies in Translation, Connectivity and Shortest Path Xinnan Dai Qihao Wen Yifei Shen Hongzhi Wen Dongsheng Li Jiliang Tang Caihua Shan LRM 47 3 0 18 Aug 2024
Transformers Can Do Arithmetic with the Right Embeddings Sean McLeish Arpit Bansal Alex Stein Neel Jain John Kirchenbauer ... B. Kailkhura A. Bhatele Jonas Geiping Avi Schwarzschild Tom Goldstein 36 28 0 27 May 2024
Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Tasks Jongho Park Jaeseung Park Zheyang Xiong Nayoung Lee Jaewoong Cho Samet Oymak Kangwook Lee Dimitris Papailiopoulos 19 69 0 06 Feb 2024
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason W. Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter F. Xia Ed H. Chi Quoc Le Denny Zhou LM&Ro LRM AI4CE ReLM 315 8,261 0 28 Jan 2022