Out-of-distribution generalization via composition: a lens through induction heads in Transformers

Proceedings of the National Academy of Sciences of the United States of America (PNAS), 2024

31 December 2024

Papers citing "Out-of-distribution generalization via composition: a lens through induction heads in Transformers"

49 / 99 papers shown

Inference-Time Intervention: Eliciting Truthful Answers from a Language ModelNeural Information Processing Systems (NeurIPS), 2023

746

833

06 Jun 2023

The Impact of Positional Encoding on Length Generalization in TransformersNeural Information Processing Systems (NeurIPS), 2023

Amirhossein Kazemnejad

Inkit Padhi

Karthikeyan N. Ramamurthy

Payel Das

Siva Reddy

359

304

31 May 2023

Faith and Fate: Limits of Transformers on CompositionalityNeural Information Processing Systems (NeurIPS), 2023

Xiang Lorraine Li

...

Xiang Ren

Yejin Choi

525

497

29 May 2023

Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD ExamplesNeural Information Processing Systems (NeurIPS), 2023

388

124

24 May 2023

Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasoners

300

24 May 2023

What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task LearningAnnual Meeting of the Association for Computational Linguistics (ACL), 2023

201

161

16 May 2023

LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language ModelsConference on Empirical Methods in Natural Language Processing (EMNLP), 2023

Lei Wang

312

382

04 Apr 2023

Pythia: A Suite for Analyzing Large Language Models Across Training and ScalingInternational Conference on Machine Learning (ICML), 2023

...

391

1,629

03 Apr 2023

GPT-4 Technical Report

...

4.6K

20,902

15 Mar 2023

A Theory of Emergent In-Context Learning as Implicit Structure Induction

Michael Hahn

Navin Goyal

LRM

257

101

14 Mar 2023

Multitask Prompt Tuning Enables Parameter-Efficient Transfer LearningInternational Conference on Learning Representations (ICLR), 2023

Huan Sun

223

151

06 Mar 2023

LLaMA: Open and Efficient Foundation Language Models

...

6.2K

17,759

27 Feb 2023

Do Deep Neural Networks Capture Compositionality in Arithmetic Reasoning?Conference of the European Chapter of the Association for Computational Linguistics (EACL), 2023

Keisuke Sakaguchi

233

15 Feb 2023

Generalization on the Unseen, Logic Reasoning and Degree CurriculumInternational Conference on Machine Learning (ICML), 2023

402

30 Jan 2023

Progress measures for grokking via mechanistic interpretabilityInternational Conference on Learning Representations (ICLR), 2023

501

626

12 Jan 2023

Transformers learn in-context by gradient descentInternational Conference on Machine Learning (ICML), 2022

493

642

15 Dec 2022

Editing Models with Task ArithmeticInternational Conference on Learning Representations (ICLR), 2022

1.2K

740

08 Dec 2022

What learning algorithm is in-context learning? Investigations with linear modelsInternational Conference on Learning Representations (ICLR), 2022

531

612

28 Nov 2022

Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallInternational Conference on Learning Representations (ICLR), 2022

620

782

01 Nov 2022

Omnigrok: Grokking Beyond Algorithmic DataInternational Conference on Learning Representations (ICLR), 2022

Ziming Liu

Eric J. Michaud

Max Tegmark

364

111

03 Oct 2022

In-context Learning and Induction Heads

...

607

700

24 Sep 2022

Toy Models of Superposition

...

1.4K

561

21 Sep 2022

What Can Transformers Learn In-Context? A Case Study of Simple Function ClassesNeural Information Processing Systems (NeurIPS), 2022

645

669

01 Aug 2022

Exploring Length Generalization in Large Language ModelsNeural Information Processing Systems (NeurIPS), 2022

348

211

11 Jul 2022

Emergent Abilities of Large Language Models

...

Tatsunori Hashimoto

532

3,128

15 Jun 2022

Unveiling Transformers with LEGO: a synthetic reasoning task

417

101

09 Jun 2022

Large Language Models are Zero-Shot ReasonersNeural Information Processing Systems (NeurIPS), 2022

1.4K

6,087

24 May 2022

Training language models to follow instructions with human feedbackNeural Information Processing Systems (NeurIPS), 2022

Carroll L. Wainwright

...

2.1K

17,638

04 Mar 2022

Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022

Luke Zettlemoyer

508

1,814

25 Feb 2022

Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsNeural Information Processing Systems (NeurIPS), 2022

2.3K

14,608

28 Jan 2022

Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets

354

487

06 Jan 2022

An Explanation of In-context Learning as Implicit Bayesian InferenceInternational Conference on Learning Representations (ICLR), 2021

1.0K

928

03 Nov 2021

MetaICL: Learning to Learn In ContextNorth American Chapter of the Association for Computational Linguistics (NAACL), 2021

Luke Zettlemoyer

682

575

29 Oct 2021

Training Verifiers to Solve Math Word Problems

...

1.1K

6,810

27 Oct 2021

Meta-learning via Language Model In-context Tuning

Yanda Chen

Ruiqi Zhong

Sheng Zha

George Karypis

He He

554

193

15 Oct 2021

RoFormer: Enhanced Transformer with Rotary Position Embedding

832

3,962

20 Apr 2021

Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order SensitivityAnnual Meeting of the Association for Computational Linguistics (ACL), 2021

948

1,383

18 Apr 2021

Surface Form Competition: Why the Highest Probability Answer Isn't Always RightConference on Empirical Methods in Natural Language Processing (EMNLP), 2021

Ari Holtzman

Peter West

Vered Schwartz

Yejin Choi

Luke Zettlemoyer

LRM

700

266

16 Apr 2021

Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factorsWorkshop on Knowledge Extraction and Integration for Deep Learning Architectures; Deep Learning Inside Out (DEELIO), 2021

282

109

29 Mar 2021

Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm

Laria Reynolds

Kyle McDonell

557

1,168

15 Feb 2021

Transformer Feed-Forward Layers Are Key-Value MemoriesConference on Empirical Methods in Natural Language Processing (EMNLP), 2020

620

1,150

29 Dec 2020

Language Models are Few-Shot LearnersNeural Information Processing Systems (NeurIPS), 2020

...

2.0K

52,526

28 May 2020

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Luke Zettlemoyer

4.0K

27,917

26 Jul 2019

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

2.9K

108,326

11 Oct 2018

Attention Is All You NeedNeural Information Processing Systems (NeurIPS), 2017

4.2K

161,538

12 Jun 2017

Linear Algebraic Structure of Word Senses, with Applications to Polysemy

Andrej Risteski

572

312

14 Jan 2016

Show and Tell: A Neural Image Caption GeneratorComputer Vision and Pattern Recognition (CVPR), 2014

606

6,373

17 Nov 2014

Neural Machine Translation by Jointly Learning to Align and TranslateInternational Conference on Learning Representations (ICLR), 2014

1.6K

28,687

01 Sep 2014

Efficient Estimation of Word Representations in Vector SpaceInternational Conference on Learning Representations (ICLR), 2013

1.4K

33,450

16 Jan 2013