Pre-Training to Learn in Context

Annual Meeting of the Association for Computational Linguistics (ACL), 2023

16 May 2023

ArXiv (abs)PDF HTML HuggingFace (2 upvotes)Github (108★)

Papers citing "Pre-Training to Learn in Context"

35 / 35 papers shown

Title
Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls Feiyang Kang Newsha Ardalani Michael Kuchnik Youssef Emad Mostafa Elhoushi Shubhabrata Sengupta Shang-Wen Li Ramya Raghavendra R. Jia Carole-Jean Wu SyDa 104 0 0 02 Oct 2025
What Matters More For In-Context Learning under Matched Compute Budgets: Pretraining on Natural Text or Incorporating Targeted Synthetic Examples? Mohammed Sabry Anya Belz 63 0 0 26 Sep 2025
RICL: Adding In-Context Adaptability to Pre-Trained Vision-Language-Action Models Kaustubh Sridhar Souradeep Dutta Dinesh Jayaraman Insup Lee LM&Ro VLM 80 5 0 04 Aug 2025
DICE: Dynamic In-Context Example Selection in LLM Agents via Efficient Knowledge Transfer Ruoyu Wang Junda Wu Yu Xia Tong Yu Ryan Rossi Julian McAuley Lina Yao 141 1 0 31 Jul 2025
Basic Reading DistillationAnnual Meeting of the Association for Computational Linguistics (ACL), 2025 Zhi Zhou Sirui Miao Xiangyu Duan Hao Yang M. Zhang 105 0 0 26 Jul 2025
Mechanistic Fine-tuning for In-context Learning Hakaze Cho Peng Luo Mariko Kato Rin Kaenbyou Naoya Inoue 281 0 0 20 May 2025
MateICL: Mitigating Attention Dispersion in Large-Scale In-Context Learning Murtadha Ahmed Wenbo Liu yunfeng 181 0 0 02 May 2025
CPRM: A LLM-based Continual Pre-training Framework for Relevance Modeling in Commercial SearchNorth American Chapter of the Association for Computational Linguistics (NAACL), 2024 Hai Ye Yixin Ji Ziyang Chen Qiang Wang Cunxiang Wang ... Jia Xu Zhongyi Liu Jinjie Gu Yuan Zhou Linjian Mo KELM CLL 511 1 0 02 Dec 2024
RARe: Retrieval Augmented Retrieval with In-Context Examples Atula Tejaswi Yoonsang Lee Sujay Sanghavi Eunsol Choi RALM LRM 152 2 0 26 Oct 2024
MiniPLM: Knowledge Distillation for Pre-Training Language ModelsInternational Conference on Learning Representations (ICLR), 2024 Yuxian Gu Hao Zhou Fandong Meng Jie Zhou Shiyu Huang 398 13 0 22 Oct 2024
Data Selection via Optimal Control for Language ModelsInternational Conference on Learning Representations (ICLR), 2024 Yuxian Gu Li Dong Hongning Wang Y. Hao Qingxiu Dong Furu Wei Minlie Huang AI4CE 299 23 0 09 Oct 2024
Wrong-of-Thought: An Integrated Reasoning Framework with Multi-Perspective Verification and Wrong InformationConference on Empirical Methods in Natural Language Processing (EMNLP), 2024 Yongheng Zhang Qiguang Chen Jingxuan Zhou Peng Wang Jiasheng Si Jin Wang Wenpeng Lu Libo Qin LRM 299 11 0 06 Oct 2024
Accelerating Inference of Networks in the Frequency DomainACM Multimedia Asia (MMAsia), 2024 Chenqiu Zhao Guanfang Dong Anup Basu 260 48 0 06 Oct 2024
FabGPT: An Efficient Large Multimodal Model for Complex Wafer Defect Knowledge Queries Yuqi Jiang Xudong Lu Qian Jin Qi Sun Hanming Wu Cheng Zhuo 269 14 0 15 Jul 2024
Token-based Decision Criteria Are Suboptimal in In-context Learning Hakaze Cho Yoshihiro Sakai Mariko Kato Kenshiro Tanaka Akira Ishii Naoya Inoue 423 6 0 24 Jun 2024
Instruction Pre-Training: Language Models are Supervised Multitask Learners Daixuan Cheng Yuxian Gu Shaohan Huang Junyu Bi Shiyu Huang Furu Wei SyDa 273 51 0 20 Jun 2024
XLand-100B: A Large-Scale Multi-Task Dataset for In-Context Reinforcement Learning Alexander Nikulin Ilya Zisman Alexey Zemtsov Viacheslav Sinii 423 11 0 13 Jun 2024
On the Noise Robustness of In-Context Learning for Text Generation Hongfu Gao Feipeng Zhang Wenyu Jiang Jun Shu Feng Zheng Jianguo Huang 258 10 0 27 May 2024
Mixture of In-Context Prompters for Tabular PFNs Derek Xu Olcay Cirit Reza Asadi Luke Huan Wei Wang 214 19 0 25 May 2024
Naive Bayes-based Context Extension for Large Language Models Jianlin Su Murtadha Ahmed Wenbo Luo Abhishek Rao Denny Zhou Hyeontaek Lim 154 8 0 26 Mar 2024
ProtLLM: An Interleaved Protein-Language LLM with Protein-as-Word Pre-Training Le Zhuo Zewen Chi Minghao Xu Heyan Huang Heqi Zheng Conghui He Xian-Ling Mao Wentao Zhang 230 18 0 28 Feb 2024
Analysing The Impact of Sequence Composition on Language Model Pre-Training Yu Zhao Yuanbin Qu Konrad Staniszewski Szymon Tworkowski Wei Liu Piotr Milo's Yuxiang Wu Pasquale Minervini 189 20 0 21 Feb 2024
Context-Former: Stitching via Latent Conditioned Sequence Modeling Ziqi Zhang Jingzehua Xu Jinxin Liu Zifeng Zhuang Xuetao Zhang Miao Liu Shuai Zhang OffRL 190 4 0 29 Jan 2024
Structured Packing in LLM Training Improves Long Context Utilization Konrad Staniszewski Szymon Tworkowski Sebastian Jaszczur Yu Zhao Henryk Michalewski Lukasz Kuciñski Piotr Milo's 298 16 0 28 Dec 2023
Beyond Output Matching: Bidirectional Alignment for Enhanced In-Context Learning Chengwei Qin Wenhan Xia Fangkai Jiao Chen Chen Yuchen Hu Bosheng Ding R. Chen Shafiq Joty 287 7 0 28 Dec 2023
Supervised Knowledge Makes Large Language Models Better In-context Learners Linyi Yang Shuibai Zhang Zhuohao Yu Guangsheng Bao Yidong Wang ... Ruochen Xu Weirong Ye Xing Xie Weizhu Chen Yue Zhang 308 25 0 26 Dec 2023
Large Language Models are Miscalibrated In-Context Learners Chengzu Li Han Zhou Goran Glavaš Anna Korhonen Ivan Vulić 156 11 0 21 Dec 2023
Simple and Effective Input Reformulations for TranslationConference on Empirical Methods in Natural Language Processing (EMNLP), 2023 Brian Yu Abby Bertics Kurt Keutzer 184 0 0 12 Nov 2023
Exploring In-Context Learning of Textless Speech Language Model for Speech Classification TasksInterspeech (Interspeech), 2023 Ming-Hao Hsu Kai-Wei Chang Shang-Wen Li Hung-yi Lee 162 9 0 19 Oct 2023
In-context Pretraining: Language Modeling Beyond Document BoundariesInternational Conference on Learning Representations (ICLR), 2023 Weijia Shi Sewon Min Maria Lomeli Chunting Zhou Margaret Li ... Victoria Lin Noah A. Smith Luke Zettlemoyer Scott Yih Mike Lewis LRM RALM SyDa 286 79 0 16 Oct 2023
Adapting Large Language Models via Reading Comprehension Daixuan Cheng Shaohan Huang Furu Wei CLL SyDa AI4CE 247 44 0 18 Sep 2023
Baby's CoThought: Leveraging Large Language Models for Enhanced Reasoning in Compact Models Zheyu Zhang Han Yang Bolei Ma David Rügamer Ercong Nie LRM 187 8 0 03 Aug 2023
Understanding In-Context Learning via Supportive Pretraining DataAnnual Meeting of the Association for Computational Linguistics (ACL), 2023 Xiaochuang Han Daniel Simig Todor Mihaylov Yulia Tsvetkov Asli Celikyilmaz Tianlu Wang AIMat 206 46 0 26 Jun 2023
A Survey on In-context LearningConference on Empirical Methods in Natural Language Processing (EMNLP), 2022 Qingxiu Dong Lei Li Damai Dai Ce Zheng Jingyuan Ma ... Zhiyong Wu Baobao Chang Xu Sun Lei Li Zhifang Sui ReLM AIMat 384 818 0 31 Dec 2022
Billion-scale similarity search with GPUsIEEE Transactions on Big Data (TBD), 2017 Jeff Johnson Matthijs Douze Edouard Grave 801 4,389 0 28 Feb 2017

All Papers

Pre-Training to Learn in Context

Papers citing "Pre-Training to Learn in Context"