The Inductive Bias of In-Context Learning: Rethinking Pretraining Example Design

9 October 2021

Papers citing "The Inductive Bias of In-Context Learning: Rethinking Pretraining Example Design"

33 / 33 papers shown

Title
Trillion 7B Technical Report Sungjun Han Juyoung Suk Suyeong An Hyungguk Kim Kyuseok Kim Wonsuk Yang Seungtaek Choi Jamin Shin 99 0 0 21 Apr 2025
Uncertainty in Action: Confidence Elicitation in Embodied Agents Tianjiao Yu Vedant Shah Muntasir Wahed Kiet A. Nguyen Adheesh Sunil Juvekar Tal August Ismini Lourentzou 40 0 0 13 Mar 2025
HierPromptLM: A Pure PLM-based Framework for Representation Learning on Heterogeneous Text-rich Networks Q. Zhu Liang Zhang Qianxiong Xu Cheng Long 134 1 0 22 Jan 2025
NExtLong: Toward Effective Long-Context Training without Long Documents Chaochen Gao Xing Wu Zijia Lin Debing Zhang Songlin Hu SyDa 64 1 0 22 Jan 2025
What Matters for In-Context Learning: A Balancing Act of Look-up and In-Weight Learning Jelena Bratulić Sudhanshu Mittal Christian Rupprecht Thomas Brox 38 0 0 09 Jan 2025
DataSculpt: Crafting Data Landscapes for Long-Context LLMs through Multi-Objective Partitioning Keer Lu Xiaonan Nie Zheng Liang Da Pan Shusen Zhang ... Weipeng Chen Zenan Zhou Guosheng Dong Bin Cui Wentao Zhang 27 0 0 02 Sep 2024
Quest: Query-centric Data Synthesis Approach for Long-context Scaling of Large Language Model Chaochen Gao Xing Wu Qingfang Fu Songlin Hu SyDa 34 5 0 30 May 2024
The Role of Learning Algorithms in Collective Action Omri Ben-Dov Jake Fawkes Samira Samadi Amartya Sanyal 24 3 0 10 May 2024
Memory Sharing for Large Language Model based Agents Hang Gao Yongfeng Zhang LLMAG 36 7 0 15 Apr 2024
Analysing The Impact of Sequence Composition on Language Model Pre-Training Yu Zhao Yuanbin Qu Konrad Staniszewski Szymon Tworkowski Wei Liu Piotr Milo's Yuxiang Wu Pasquale Minervini 34 14 0 21 Feb 2024
Improving Domain Adaptation through Extended-Text Reading Comprehension Ting Jiang Shaohan Huang Shengyue Luo Zihan Zhang Haizhen Huang ... Weiwei Deng Feng Sun Qi Zhang Deqing Wang Fuzhen Zhuang AI4CE 35 10 0 14 Jan 2024
Structured Packing in LLM Training Improves Long Context Utilization Konrad Staniszewski Szymon Tworkowski Sebastian Jaszczur Yu Zhao Henryk Michalewski Lukasz Kuciñski Piotr Milo's 41 13 0 28 Dec 2023
User Modeling in the Era of Large Language Models: Current Research and Future Directions Zhaoxuan Tan Meng-Long Jiang 18 8 0 11 Dec 2023
In-Context Learning Creates Task Vectors Roee Hendel Mor Geva Amir Globerson 18 131 0 24 Oct 2023
Pretraining Language Models with Text-Attributed Heterogeneous Graphs Tao Zou Le Yu Yifei Huang Leilei Sun Bo Du AI4CE 17 17 0 19 Oct 2023
In-context Pretraining: Language Modeling Beyond Document Boundaries Weijia Shi Sewon Min Maria Lomeli Chunting Zhou Margaret Li ... Victoria Lin Noah A. Smith Luke Zettlemoyer Scott Yih Mike Lewis LRM RALM SyDa 27 48 0 16 Oct 2023
When Large Language Models Meet Citation: A Survey Yang Zhang Yufei Wang Kai Wang Quan Z. Sheng Lina Yao A. Mahmood Wei Emma Zhang Rongying Zhao 45 7 0 18 Sep 2023
Pre-Training to Learn in Context Yuxian Gu Li Dong Furu Wei Minlie Huang CLIP LRM ReLM 108 37 0 16 May 2023
The Learnability of In-Context Learning Noam Wies Yoav Levine Amnon Shashua 117 91 0 14 Mar 2023
Model-tuning Via Prompts Makes NLP Models Adversarially Robust Mrigank Raman Pratyush Maini J. Zico Kolter Zachary Chase Lipton Danish Pruthi AAML 25 17 0 13 Mar 2023
Multiple Thinking Achieving Meta-Ability Decoupling for Object Navigation Ronghao Dang Lu Chen Liuyi Wang Zongtao He Chengju Liu Qi Chen LRM 19 8 0 03 Feb 2023
In-Context Retrieval-Augmented Language Models Ori Ram Yoav Levine Itay Dalmedigos Dor Muhlgay Amnon Shashua Kevin Leyton-Brown Y. Shoham KELM RALM LRM 15 536 0 31 Jan 2023
On the Ability of Graph Neural Networks to Model Interactions Between Vertices Noam Razin Tom Verbin Nadav Cohen 19 10 0 29 Nov 2022
Downstream Datasets Make Surprisingly Good Pretraining Corpora Kundan Krishna Saurabh Garg Jeffrey P. Bigham Zachary Chase Lipton 38 30 0 28 Sep 2022
Transformer Vs. MLP-Mixer: Exponential Expressive Gap For NLP Problems D. Navon A. Bronstein MoE 36 0 0 17 Aug 2022
Standing on the Shoulders of Giant Frozen Language Models Yoav Levine Itay Dalmedigos Ori Ram Yoel Zeldes Daniel Jannai ... Barak Lenz Shai Shalev-Shwartz Amnon Shashua Kevin Leyton-Brown Y. Shoham VLM 19 49 0 21 Apr 2022
LinkBERT: Pretraining Language Models with Document Links Michihiro Yasunaga J. Leskovec Percy Liang KELM 12 352 0 29 Mar 2022
Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networks Noam Razin Asaf Maman Nadav Cohen 33 29 0 27 Jan 2022
Learning To Retrieve Prompts for In-Context Learning Ohad Rubin Jonathan Herzig Jonathan Berant VPVLM RALM 14 665 0 16 Dec 2021
Structure Inducing Pre-Training Matthew B. A. McDermott Brendan Yap Peter Szolovits Marinka Zitnik 30 18 0 18 Mar 2021
Making Pre-trained Language Models Better Few-shot Learners Tianyu Gao Adam Fisch Danqi Chen 241 1,916 0 31 Dec 2020
Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks Nandan Thakur Nils Reimers Johannes Daxenberger Iryna Gurevych 200 241 0 16 Oct 2020
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 294 6,950 0 20 Apr 2018