Impact of Pretraining Term Frequencies on Few-Shot Reasoning

15 February 2022

Papers citing "Impact of Pretraining Term Frequencies on Few-Shot Reasoning"

41 / 41 papers shown

Title
ICL CIPHERS: Quantifying "Learning'' in In-Context Learning via Substitution Ciphers Zhouxiang Fang Aayush Mishra Muhan Gao Anqi Liu Daniel Khashabi 44 0 0 28 Apr 2025
Mitigating Copy Bias in In-Context Learning through Neuron Pruning Ameen Ali Lior Wolf Ivan Titov 36 2 0 02 Oct 2024
Explicitly Encoding Structural Symmetry is Key to Length Generalization in Arithmetic Tasks Mahdi Sabbaghi George Pappas Hamed Hassani Surbhi Goel 36 4 0 04 Jun 2024
A Glitch in the Matrix? Locating and Detecting Language Model Grounding with Fakepedia Giovanni Monea Maxime Peyrard Martin Josifoski Vishrav Chaudhary Jason Eisner Emre Kiciman Hamid Palangi Barun Patra Robert West KELM 51 12 0 04 Dec 2023
Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Tasks Rahul Ramesh Ekdeep Singh Lubana Mikail Khona Robert P. Dick Hidenori Tanaka CoGe 33 6 0 21 Nov 2023
ICXML: An In-Context Learning Framework for Zero-Shot Extreme Multi-Label Classification Yaxin Zhu Hamed Zamani 38 3 0 16 Nov 2023
GPT4AIGChip: Towards Next-Generation AI Accelerator Design Automation via Large Language Models Yonggan Fu Yongan Zhang Zhongzhi Yu Sixu Li Zhifan Ye Chaojian Li Cheng Wan Ying Lin 40 60 0 19 Sep 2023
In-Context Learning Learns Label Relationships but Is Not Conventional Learning Jannik Kossen Y. Gal Tom Rainforth 32 27 0 23 Jul 2023
Self-ICL: Zero-Shot In-Context Learning with Self-Generated Demonstrations Wei-Lin Chen Cheng-Kuang Wu Yun-Nung Chen Hsin-Hsi Chen 18 27 0 24 May 2023
A Latent Space Theory for Emergent Abilities in Large Language Models Hui Jiang LRM 23 35 0 19 Apr 2023
Koala: An Index for Quantifying Overlaps with Pre-training Corpora Thuy-Trang Vu Xuanli He Gholamreza Haffari Ehsan Shareghi CLL 21 12 0 26 Mar 2023
In-context Example Selection with Influences Nguyen Tai Eric Wong 11 48 0 21 Feb 2023
Towards Reasoning in Large Language Models: A Survey Jie Huang Kevin Chen-Chuan Chang LM&MA ELM LRM 27 582 0 20 Dec 2022
Z-ICL: Zero-Shot In-Context Learning with Pseudo-Demonstrations Xinxi Lyu Sewon Min Iz Beltagy Luke Zettlemoyer Hannaneh Hajishirzi VLM 14 62 0 19 Dec 2022
Demystifying Prompts in Language Models via Perplexity Estimation Hila Gonen Srini Iyer Terra Blevins Noah A. Smith Luke Zettlemoyer LRM 27 195 0 08 Dec 2022
Large Language Models Struggle to Learn Long-Tail Knowledge Nikhil Kandpal H. Deng Adam Roberts Eric Wallace Colin Raffel RALM KELM 36 381 0 15 Nov 2022
Logical Tasks for Measuring Extrapolation and Rule Comprehension Ippei Fujisawa Ryota Kanai ELM LRM 25 4 0 14 Nov 2022
Complex Reading Comprehension Through Question Decomposition Xiao-Yu Guo Yuan-Fang Li Gholamreza Haffari ReLM 11 8 0 07 Nov 2022
A Simple, Yet Effective Approach to Finding Biases in Code Generation Spyridon Mouselinos Mateusz Malinowski Henryk Michalewski 10 7 0 31 Oct 2022
Transfer Learning with Synthetic Corpora for Spatial Role Labeling and Reasoning Roshanak Mirzaee Parisa Kordjamshidi SyDa LRM 18 33 0 30 Oct 2022
Towards Relation Extraction From Speech Tongtong Wu Guitao Wang Jinming Zhao Zhaoran Liu Guilin Qi Yuan-Fang Li Gholamreza Haffari 29 11 0 17 Oct 2022
Mind's Eye: Grounded Language Model Reasoning through Simulation Ruibo Liu Jason W. Wei S. Gu Te-Yen Wu Soroush Vosoughi Claire Cui Denny Zhou Andrew M. Dai ReLM LRM 116 79 0 11 Oct 2022
Reflection of Thought: Inversely Eliciting Numerical Reasoning in Language Models via Solving Linear Systems Fan Zhou Haoyu Dong Qian Liu Zhoujun Cheng Shi Han Dongmei Zhang ReLM LRM 31 5 0 11 Oct 2022
State-of-the-art generalisation research in NLP: A taxonomy and review Dieuwke Hupkes Mario Giulianelli Verna Dankers Mikel Artetxe Yanai Elazar ... Leila Khalatbari Maria Ryskina Rita Frieske Ryan Cotterell Zhijing Jin 114 93 0 06 Oct 2022
ChemAlgebra: Algebraic Reasoning on Chemical Reactions Andrea Valenti D. Bacciu Antonio Vergari OOD LRM 32 0 0 05 Oct 2022
Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought Abulhair Saparov He He ELM LRM ReLM 121 275 0 03 Oct 2022
Systematic Generalization and Emergent Structures in Transformers Trained on Structured Tasks Yuxuan Li James L. McClelland 36 17 0 02 Oct 2022
What Can Transformers Learn In-Context? A Case Study of Simple Function Classes Shivam Garg Dimitris Tsipras Percy Liang Gregory Valiant 21 447 0 01 Aug 2022
Language models show human-like content effects on reasoning tasks Ishita Dasgupta Andrew Kyle Lampinen Stephanie C. Y. Chan Hannah R. Sheahan Antonia Creswell D. Kumaran James L. McClelland Felix Hill ReLM LRM 28 180 0 14 Jul 2022
Exploring Length Generalization in Large Language Models Cem Anil Yuhuai Wu Anders Andreassen Aitor Lewkowycz Vedant Misra V. Ramasesh Ambrose Slone Guy Gur-Ari Ethan Dyer Behnam Neyshabur ReLM LRM 33 158 0 11 Jul 2022
Emergent Abilities of Large Language Models Jason W. Wei Yi Tay Rishi Bommasani Colin Raffel Barret Zoph ... Tatsunori Hashimoto Oriol Vinyals Percy Liang J. Dean W. Fedus ELM ReLM LRM 48 2,333 0 15 Jun 2022
Prototypical Calibration for Few-shot Learning of Language Models Zhixiong Han Y. Hao Li Dong Yutao Sun Furu Wei 170 52 0 20 May 2022
Data Distributional Properties Drive Emergent In-Context Learning in Transformers Stephanie C. Y. Chan Adam Santoro Andrew Kyle Lampinen Jane X. Wang Aaditya K. Singh Pierre Harvey Richemond J. Mcclelland Felix Hill 50 244 0 22 Apr 2022
GPT-NeoX-20B: An Open-Source Autoregressive Language Model Sid Black Stella Biderman Eric Hallahan Quentin G. Anthony Leo Gao ... Shivanshu Purohit Laria Reynolds J. Tow Benqi Wang Samuel Weinbach 63 800 0 14 Apr 2022
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? Sewon Min Xinxi Lyu Ari Holtzman Mikel Artetxe M. Lewis Hannaneh Hajishirzi Luke Zettlemoyer LLMAG LRM 23 1,395 0 25 Feb 2022
Frequency Effects on Syntactic Rule Learning in Transformers Jason W. Wei Dan Garrette Tal Linzen Ellie Pavlick 85 62 0 14 Sep 2021
Competency Problems: On Finding and Removing Artifacts in Language Data Matt Gardner William Merrill Jesse Dodge Matthew E. Peters Alexis Ross Sameer Singh Noah A. Smith 166 107 0 17 Apr 2021
Memorization vs. Generalization: Quantifying Data Leakage in NLP Performance Evaluation Aparna Elangovan Jiayuan He Karin Verspoor TDI FedML 164 89 0 03 Feb 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 253 1,989 0 31 Dec 2020
Language Models as Knowledge Bases? Fabio Petroni Tim Rocktaschel Patrick Lewis A. Bakhtin Yuxiang Wu Alexander H. Miller Sebastian Riedel KELM AI4MH 415 2,584 0 03 Sep 2019
Hypothesis Only Baselines in Natural Language Inference Adam Poliak Jason Naradowsky Aparajita Haldar Rachel Rudinger Benjamin Van Durme 190 576 0 02 May 2018