Understanding In-Context Learning via Supportive Pretraining Data

26 June 2023

Papers citing "Understanding In-Context Learning via Supportive Pretraining Data"

28 / 28 papers shown

Title
Scaling sparse feature circuit finding for in-context learning Dmitrii Kharlapenko Shivalika Singh Fazl Barez Arthur Conmy Neel Nanda 26 0 0 18 Apr 2025
BPO: Towards Balanced Preference Optimization between Knowledge Breadth and Depth in Alignment Sizhe Wang Yongqi Tong Hengyuan Zhang Dawei Li Xin Zhang Tianlong Chen 85 5 0 21 Feb 2025
What Matters for In-Context Learning: A Balancing Act of Look-up and In-Weight Learning Jelena Bratulić Sudhanshu Mittal Christian Rupprecht Thomas Brox 41 1 0 09 Jan 2025
Scalable Influence and Fact Tracing for Large Language Model Pretraining Tyler A. Chang Dheeraj Rajagopal Tolga Bolukbasi Lucas Dixon Ian Tenney TDI 35 2 0 22 Oct 2024
Influential Language Data Selection via Gradient Trajectory Pursuit Zhiwei Deng Tao Li Yang Li 26 1 0 22 Oct 2024
Wrong-of-Thought: An Integrated Reasoning Framework with Multi-Perspective Verification and Wrong Information Yongheng Zhang Qiguang Chen Jingxuan Zhou Peng Wang Jiasheng Si Jin Wang Wenpeng Lu Libo Qin LRM 49 3 0 06 Oct 2024
No Size Fits All: The Perils and Pitfalls of Leveraging LLMs Vary with Company Size Ashok Urlana Charaka Vinayak Kumar B. Garlapati Ajeet Kumar Singh Rahul Mishra 37 1 0 21 Jul 2024
Token-based Decision Criteria Are Suboptimal in In-context Learning Hakaze Cho Yoshihiro Sakai Mariko Kato Kenshiro Tanaka Akira Ishii Naoya Inoue 46 2 0 24 Jun 2024
Do Language Models Exhibit Human-like Structural Priming Effects? Jaap Jumelet Willem H. Zuidema Arabella J. Sinclair 46 5 0 07 Jun 2024
From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examples Robert Vacareanu Vlad-Andrei Negru Vasile Suciu Mihai Surdeanu 31 28 0 11 Apr 2024
Can large language models explore in-context? Akshay Krishnamurthy Keegan Harris Dylan J. Foster Cyril Zhang Aleksandrs Slivkins LM&Ro LLMAG LRM 126 23 0 22 Mar 2024
Unveiling the Generalization Power of Fine-Tuned Large Language Models Haoran Yang Yumeng Zhang Jiaqi Xu Hongyuan Lu Pheng Ann Heng Wai Lam 45 29 0 14 Mar 2024
Analyzing and Adapting Large Language Models for Few-Shot Multilingual NLU: Are We There Yet? E. Razumovskaia Ivan Vulić Anna Korhonen 46 6 0 04 Mar 2024
NoisyICL: A Little Noise in Model Parameters Calibrates In-context Learning Yufeng Zhao Yoshihiro Sakai Naoya Inoue 33 3 0 08 Feb 2024
LESS: Selecting Influential Data for Targeted Instruction Tuning Mengzhou Xia Sadhika Malladi Suchin Gururangan Sanjeev Arora Danqi Chen 82 186 0 06 Feb 2024
Unlearning Traces the Influential Training Data of Language Models Masaru Isonuma Ivan Titov MU 29 6 0 26 Jan 2024
Supervised Knowledge Makes Large Language Models Better In-context Learners Linyi Yang Shuibai Zhang Zhuohao Yu Guangsheng Bao Yidong Wang ... Ruochen Xu Weirong Ye Xing Xie Weizhu Chen Yue Zhang 21 14 0 26 Dec 2023
MM-Narrator: Narrating Long-form Videos with Multimodal In-Context Learning Chaoyi Zhang K. Lin Zhengyuan Yang Jianfeng Wang Linjie Li Chung-Ching Lin Zicheng Liu Lijuan Wang VGen 21 28 0 29 Nov 2023
Do pretrained Transformers Learn In-Context by Gradient Descent? Lingfeng Shen Aayush Mishra Daniel Khashabi 39 7 0 12 Oct 2023
SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore Sewon Min Suchin Gururangan Eric Wallace Hannaneh Hajishirzi Noah A. Smith Luke Zettlemoyer AILaw 22 63 0 08 Aug 2023
Pre-Training to Learn in Context Yuxian Gu Li Dong Furu Wei Minlie Huang CLIP LRM ReLM 116 37 0 16 May 2023
Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond Jingfeng Yang Hongye Jin Ruixiang Tang Xiaotian Han Qizhang Feng Haoming Jiang Bing Yin Xia Hu LM&MA 137 622 0 26 Apr 2023
Simfluence: Modeling the Influence of Individual Training Examples by Simulating Training Runs Kelvin Guu Albert Webson Ellie Pavlick Lucas Dixon Ian Tenney Tolga Bolukbasi TDI 70 33 0 14 Mar 2023
Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learning Xinyi Wang Wanrong Zhu Michael Stephen Saxon Mark Steyvers William Yang Wang BDL 53 91 0 27 Jan 2023
In-context Learning and Induction Heads Catherine Olsson Nelson Elhage Neel Nanda Nicholas Joseph Nova Dassarma ... Tom B. Brown Jack Clark Jared Kaplan Sam McCandlish C. Olah 250 460 0 24 Sep 2022
Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference Timo Schick Hinrich Schütze 258 1,589 0 21 Jan 2020
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 245 1,821 0 17 Sep 2019
Language Models as Knowledge Bases? Fabio Petroni Tim Rocktaschel Patrick Lewis A. Bakhtin Yuxiang Wu Alexander H. Miller Sebastian Riedel KELM AI4MH 417 2,588 0 03 Sep 2019