Understanding Transformer Reasoning Capabilities via Graph Algorithms

28 May 2024

Vahab Mirrokni

Papers citing "Understanding Transformer Reasoning Capabilities via Graph Algorithms"

29 / 29 papers shown

Title
Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction Vaishnavh Nagarajan Chen Henry Wu Charles Ding Aditi Raghunathan 28 0 0 21 Apr 2025
Provable Failure of Language Models in Learning Majority Boolean Logic via Gradient Descent Bo Chen Zhenmei Shi Zhao-quan Song Jiahao Zhang NAI LRM AI4CE 53 1 0 07 Apr 2025
Language Models, Graph Searching, and Supervision Adulteration: When More Supervision is Less and How to Make More More Arvid Frydenlund LRM 48 0 0 13 Mar 2025
Combinatorial Optimization via LLM-driven Iterated Fine-tuning Pranjal Awasthi Sreenivas Gollapudi Ravi Kumar Kamesh Munagala 63 0 0 10 Mar 2025
Depth-Width tradeoffs in Algorithmic Reasoning of Graph Tasks with Transformers Gilad Yehudai Clayton Sanford Maya Bechler-Speicher Orr Fischer Ran Gilad-Bachrach Amir Globerson 53 0 0 03 Mar 2025
Compositional Reasoning with Transformers, RNNs, and Chain of Thought Gilad Yehudai Noah Amsel Joan Bruna LRM 57 1 0 03 Mar 2025
Reasoning with Latent Thoughts: On the Power of Looped Transformers Nikunj Saunshi Nishanth Dikkala Zhiyuan Li Sanjiv Kumar Sashank J. Reddi OffRL LRM AI4CE 45 9 0 24 Feb 2025
Graph-Aware Isomorphic Attention for Adaptive Dynamics in Transformers Markus J. Buehler AI4CE 35 1 0 04 Jan 2025
Lost-in-Distance: Impact of Contextual Proximity on LLM Performance in Graph Tasks Hamed Firooz Maziar Sanjabi Wenlong Jiang Xiaoling Zhai 63 3 0 03 Jan 2025
Best of Both Worlds: Advantages of Hybrid Graph Sequence Models Ali Behrouz Ali Parviz Mahdi Karami Clayton Sanford Bryan Perozzi Vahab Mirrokni 79 2 0 23 Nov 2024
A Theory for Compressibility of Graph Transformers for Transductive Learning Hamed Shirzad Honghao Lin A. Velingker B. Venkatachalam David P. Woodruff Danica J. Sutherland 72 1 0 20 Nov 2024
A Hierarchical Language Model For Interpretable Graph Reasoning Sambhav Khurana Xiner Li Shurui Gui Shuiwang Ji LRM 31 0 0 29 Oct 2024
Mixture of Parrots: Experts improve memorization more than reasoning Samy Jelassi Clara Mohri David Brandfonbrener Alex Gu Nikhil Vyas Nikhil Anand David Alvarez-Melis Yuanzhi Li Sham Kakade Eran Malach MoE 28 4 0 24 Oct 2024
GraphTeam: Facilitating Large Language Model-based Graph Analysis via Multi-Agent Collaboration Xin Sky Li Qizhi Chu Y. Chen Yang Liu Yaoqi Liu Zekai Yu Weize Chen Chen Qian C. Shi Cheng Yang LLMAG 48 2 0 23 Oct 2024
CodeGraph: Enhancing Graph Reasoning of LLMs with Code Qiaolong Cai Zhaowei Wang Shizhe Diao James Kwok Yangqiu Song LRM 41 2 0 25 Aug 2024
Revisiting the Graph Reasoning Ability of Large Language Models: Case Studies in Translation, Connectivity and Shortest Path Xinnan Dai Qihao Wen Yifei Shen Hongzhi Wen Dongsheng Li Jiliang Tang Caihua Shan LRM 47 3 0 18 Aug 2024
Reasoning in Large Language Models: A Geometric Perspective Romain Cosentino Sarath Shekkizhar LRM 42 2 0 02 Jul 2024
Revisiting Random Walks for Learning on Graphs Jinwoo Kim Olga Zaghen Ayhan Suleymanzade Youngmin Ryou Seunghoon Hong 54 0 0 01 Jul 2024
Text-space Graph Foundation Models: Comprehensive Benchmarks and New Insights Zhikai Chen Haitao Mao Jingzhe Liu Yu Song Bingheng Li ... Bahare Fatemi Anton Tsitsulin Bryan Perozzi Hui Liu Jiliang Tang 25 10 0 15 Jun 2024
Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning Bahare Fatemi Mehran Kazemi Anton Tsitsulin Karishma Malkan Jinyeong Yim John Palowitch Sungyong Seo Jonathan J. Halcrow Bryan Perozzi LRM 35 26 0 13 Jun 2024
The CLRS-Text Algorithmic Reasoning Language Benchmark Larisa Markeeva Sean McLeish Borja Ibarz Wilfried Bounsi Olga Kozlova Alex Vitvitskyi Charles Blundell Tom Goldstein Avi Schwarzschild Petar Veličković LRM 34 12 0 06 Jun 2024
Microstructures and Accuracy of Graph Recall by Large Language Models Yanbang Wang Hejie Cui Jon M. Kleinberg 34 2 0 19 Feb 2024
GPT-4 Doesn't Know It's Wrong: An Analysis of Iterative Prompting for Reasoning Problems Kaya Stechly Matthew Marquez Subbarao Kambhampati LRM 155 84 0 19 Oct 2023
A Logic for Expressing Log-Precision Transformers William Merrill Ashish Sabharwal ReLM NAI LRM 48 46 0 06 Oct 2022
Systematic Generalization and Emergent Structures in Transformers Trained on Structured Tasks Yuxuan Li James L. McClelland 21 17 0 02 Oct 2022
The CLRS Algorithmic Reasoning Benchmark Petar Velivcković Adria Puigdomenech Badia David Budden Razvan Pascanu Andrea Banino Mikhail Dashevskiy R. Hadsell Charles Blundell 157 86 0 31 May 2022
Large Language Models are Zero-Shot Reasoners Takeshi Kojima S. Gu Machel Reid Yutaka Matsuo Yusuke Iwasawa ReLM LRM 291 4,048 0 24 May 2022
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason W. Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter F. Xia Ed H. Chi Quoc Le Denny Zhou LM&Ro LRM AI4CE ReLM 315 8,261 0 28 Jan 2022
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 226 4,424 0 23 Jan 2020