Making Transformers Solve Compositional Tasks

9 August 2021

Santiago Ontañón

Joshua Ainslie

Vaclav Cvicek

Zachary Kenneth Fisher

ArXiv PDF HTML

Papers citing "Making Transformers Solve Compositional Tasks"

50 / 56 papers shown

Title
Exploring Compositional Generalization (in ReCOGS_pos) by Transformers using Restricted Access Sequence Processing (RASP) William Bruns 38 0 0 21 Apr 2025
Learning to Substitute Components for Compositional Generalization Z. Li Gangwei Jiang Chenwang Wu Ying Wei Defu Lian Enhong Chen 57 0 0 28 Feb 2025
Data Distributional Properties As Inductive Bias for Systematic Generalization Felipe del-Rio Alain Raymond-Sáez Daniel Florea Rodrigo Toro Icarte Julio Hurtado Cristian B. Calderon Á. Soto AI4CE 33 0 0 27 Feb 2025
The Role of Sparsity for Length Generalization in Transformers Noah Golowich Samy Jelassi David Brandfonbrener Sham Kakade Eran Malach 37 0 0 24 Feb 2025
Strassen Attention: Unlocking Compositional Abilities in Transformers Based on a New Lower Bound Method A. Kozachinskiy Felipe Urrutia Hector Jimenez Tomasz Steifer Germán Pizarro Matías Fuentes Francisco Meza Cristian Buc Cristóbal Rojas 47 1 0 31 Jan 2025
Quantifying artificial intelligence through algebraic generalization Takuya Ito Murray Campbell L. Horesh Tim Klinger Parikshit Ram ELM 46 0 0 08 Nov 2024
Block-Operations: Using Modular Routing to Improve Compositional Generalization Florian Dietz Dietrich Klakow AI4CE 19 0 0 01 Aug 2024
Aligning Programming Language and Natural Language: Exploring Design Choices in Multi-Modal Transformer-Based Embedding for Bug Localization Partha Chakraborty Venkatraman Arumugam M. Nagappan 24 0 0 25 Jun 2024
MoEUT: Mixture-of-Experts Universal Transformers Róbert Csordás Kazuki Irie Jürgen Schmidhuber Christopher Potts Christopher D. Manning MoE 40 5 0 25 May 2024
SPOR: A Comprehensive and Practical Evaluation Method for Compositional Generalization in Data-to-Text Generation Ziyao Xu Houfeng Wang 16 2 0 17 May 2024
Philosophy of Cognitive Science in the Age of Deep Learning Raphaël Millière AI4CE NAI 30 3 0 07 May 2024
What makes Models Compositional? A Theoretical View: With Supplement Parikshit Ram Tim Klinger Alexander G. Gray CoGe 34 6 0 02 May 2024
Towards Understanding the Relationship between In-context Learning and Compositional Generalization Sungjun Han Sebastian Padó CoGe 21 2 0 18 Mar 2024
A Neural Rewriting System to Solve Algorithmic Problems Flavio Petruzzellis Alberto Testolin A. Sperduti NAI 34 0 0 27 Feb 2024
Benchmarking GPT-4 on Algorithmic Problems: A Systematic Evaluation of Prompting Strategies Flavio Petruzzellis Alberto Testolin A. Sperduti ELM 30 7 0 27 Feb 2024
On the generalization capacity of neural networks during generic multimodal reasoning Takuya Ito Soham Dan Mattia Rigotti James Kozloski Murray Campbell LRM 30 2 0 26 Jan 2024
A Philosophical Introduction to Language Models -- Part I: Continuity With Classic Debates Raphael Milliere Cameron Buckner LRM ELM 22 18 0 08 Jan 2024
Compositional Generalization in Spoken Language Understanding Avik Ray Yilin Shen Hongxia Jin CoGe 17 1 0 25 Dec 2023
Probabilistic Transformer: A Probabilistic Dependency Model for Contextual Word Representation Haoyi Wu Kewei Tu 93 3 0 26 Nov 2023
Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Tasks Rahul Ramesh Ekdeep Singh Lubana Mikail Khona Robert P. Dick Hidenori Tanaka CoGe 27 6 0 21 Nov 2023
Neural-Logic Human-Object Interaction Detection Liulei Li Jianan Wei Wenguan Wang Yi Yang 29 16 0 16 Nov 2023
MetaReVision: Meta-Learning with Retrieval for Visually Grounded Compositional Concept Acquisition Guangyue Xu Parisa Kordjamshidi Joyce Chai 13 2 0 02 Nov 2023
The Impact of Depth on Compositional Generalization in Transformer Language Models Jackson Petty Sjoerd van Steenkiste Ishita Dasgupta Fei Sha Daniel H Garrette Tal Linzen AI4CE VLM 10 16 0 30 Oct 2023
ExeDec: Execution Decomposition for Compositional Generalization in Neural Program Synthesis Kensen Shi Joey Hong Yinlin Deng Pengcheng Yin Manzil Zaheer Charles Sutton 18 18 0 26 Jul 2023
Teaching Arithmetic to Small Transformers Nayoung Lee Kartik K. Sreenivasan Jason D. Lee Kangwook Lee Dimitris Papailiopoulos LRM 21 81 0 07 Jul 2023
Learning to Substitute Spans towards Improving Compositional Generalization Zhaoyi Li Ying Wei Defu Lian 10 9 0 05 Jun 2023
The Impact of Positional Encoding on Length Generalization in Transformers Amirhossein Kazemnejad Inkit Padhi K. Ramamurthy Payel Das Siva Reddy 19 177 0 31 May 2023
Randomized Positional Encodings Boost Length Generalization of Transformers Anian Ruoss Grégoire Delétang Tim Genewein Jordi Grau-Moya Róbert Csordás Mehdi Abbana Bennani Shane Legg J. Veness LLMAG 25 99 0 26 May 2023
Explainable Verbal Reasoner Plus (EVR+): A Natural Language Reasoning Framework that Supports Diverse Compositional Reasoning Zhengzhong Liang Zeyu Zhang Steven Bethard Mihai Surdeanu ReLM LRM 18 1 0 28 Apr 2023
ReCOGS: How Incidental Details of a Logical Form Overshadow an Evaluation of Semantic Interpretation Zhengxuan Wu Christopher D. Manning Christopher Potts 22 22 0 24 Mar 2023
Sequential Query Encoding For Complex Query Answering on Knowledge Graphs Jiaxin Bai Tianshi Zheng Yangqiu Song 14 13 0 25 Feb 2023
Empirical Investigation of Neural Symbolic Reasoning Strategies Yoichi Aoki Keito Kudo Tatsuki Kuribayashi Ana Brassard Masashi Yoshikawa Keisuke Sakaguchi Kentaro Inui 8 2 0 16 Feb 2023
MAQA: A Multimodal QA Benchmark for Negation Judith Yue Li Aren Jansen Qingqing Huang Joonseok Lee Ravi Ganti Dima Kuzmin 25 5 0 09 Jan 2023
Uncontrolled Lexical Exposure Leads to Overestimation of Compositional Generalization in Pretrained Models Najoung Kim Tal Linzen P. Smolensky 13 30 0 21 Dec 2022
A Short Survey of Systematic Generalization Yuanpeng Li AI4CE 22 1 0 22 Nov 2022
Structural generalization is hard for sequence-to-sequence models Yuekun Yao Alexander Koller 17 21 0 24 Oct 2022
The Curious Case of Absolute Position Embeddings Koustuv Sinha Amirhossein Kazemnejad Siva Reddy J. Pineau Dieuwke Hupkes Adina Williams 75 15 0 23 Oct 2022
Categorizing Semantic Representations for Neural Machine Translation Yongjing Yin Yafu Li Fandong Meng Jie Zhou Yue Zhang 16 6 0 13 Oct 2022
Compositional Generalisation with Structured Reordering and Fertility Layers Matthias Lindemann Alexander Koller Ivan Titov CoGe 27 7 0 06 Oct 2022
State-of-the-art generalisation research in NLP: A taxonomy and review Dieuwke Hupkes Mario Giulianelli Verna Dankers Mikel Artetxe Yanai Elazar ... Leila Khalatbari Maria Ryskina Rita Frieske Ryan Cotterell Zhijing Jin 106 92 0 06 Oct 2022
Neural-Symbolic Recursive Machine for Systematic Generalization Qing Li Yixin Zhu Yitao Liang Ying Nian Wu Song-Chun Zhu Siyuan Huang NAI 24 9 0 04 Oct 2022
Systematic Generalization and Emergent Structures in Transformers Trained on Structured Tasks Yuxuan Li James L. McClelland 29 17 0 02 Oct 2022
Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing Linlu Qiu Peter Shaw Panupong Pasupat Tianze Shi Jonathan Herzig Emily Pitler Fei Sha Kristina Toutanova AI4CE LRM 25 52 0 24 May 2022
Fusing finetuned models for better pretraining Leshem Choshen Elad Venezian Noam Slonim Yoav Katz FedML AI4CE MoMe 36 86 0 06 Apr 2022
LogicInference: A New Dataset for Teaching Logical Inference to seq2seq Models Santiago Ontanon Joshua Ainslie Vaclav Cvicek Zachary Kenneth Fisher NAI ReLM LRM 11 13 0 28 Mar 2022
Revisiting the Compositional Generalization Abilities of Neural Sequence Models Arkil Patel S. Bhattamishra Phil Blunsom Navin Goyal BDL CoGe 13 32 0 14 Mar 2022
An Application of Pseudo-Log-Likelihoods to Natural Language Scoring Darren Abramson Ali Emami 38 3 0 23 Jan 2022
Improving Compositional Generalization with Latent Structure and Data Augmentation Linlu Qiu Peter Shaw Panupong Pasupat Pawel Krzysztof Nowak Tal Linzen Fei Sha Kristina Toutanova CoGe 23 57 0 14 Dec 2021
Systematic Generalization with Edge Transformers Leon Bergen Timothy J. O'Donnell Dzmitry Bahdanau 10 46 0 01 Dec 2021
The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalization Róbert Csordás Kazuki Irie Jürgen Schmidhuber AI4CE 17 55 0 14 Oct 2021