How Do Large Language Models Acquire Factual Knowledge During Pretraining?

17 June 2024

Du-Seong Chang

Minjoon Seo

KELM

ArXiv (abs)PDF HTML HuggingFace (32 upvotes)

Papers citing "How Do Large Language Models Acquire Factual Knowledge During Pretraining?"

50 / 56 papers shown

Title
Exploring the Influence of Relevant Knowledge for Natural Language Generation Interpretability Iván Martínez-Murillo Paloma Moreda Elena Lloret 34 0 0 28 Oct 2025
Generalization or Memorization: Dynamic Decoding for Mode Steering Xuanming Zhang 104 0 0 25 Oct 2025
Blackbox Model Provenance via Palimpsestic Membership Inference Rohith Kuditipudi Jing-ling Huang Sally Zhu Diyi Yang Christopher Potts Abigail Z. Jacobs 120 0 0 22 Oct 2025
Learning from the Best, Differently: A Diversity-Driven Rethinking on Data Selection Hongyi He Xiao Liu Zhenghao Lin Mingni Tang Y. Cheng Jintao Wang W. Li Peng Cheng Yeyun Gong OODD 121 0 0 21 Oct 2025
From Memorization to Generalization: Fine-Tuning Large Language Models for Biomedical Term-to-Identifier Normalization Suswitha Pericharla D. B. Hier Tayo Obafemi-Ajayi 83 0 0 21 Oct 2025
Facts in Stats: Impacts of Pretraining Diversity on Language Model Generalization Tina Behnia Puneesh Deora Christos Thrampoulidis 68 0 0 17 Oct 2025
Tracing Multilingual Knowledge Acquisition Dynamics in Domain Adaptation: A Case Study of English-Japanese Biomedical Adaptation Xin Zhao Naoki Yoshinaga Yuma Tsuta Akiko Aizawa 72 0 0 14 Oct 2025
STEAM: A Semantic-Level Knowledge Editing Framework for Large Language Models Geunyeong Jeong Juoh Sun Seonghee Lee Harksoo Kim KELM 92 0 0 12 Oct 2025
Are LLMs Better GNN Helpers? Rethinking Robust Graph Learning under Deficiencies with Iterative Refinement Zhaoyan Wang Zheng Gao Arogya Kharel In-Young Ko 56 0 0 02 Oct 2025
Train Once, Answer All: Many Pretraining Experiments for the Cost of One Sebastian Bordt Martin Pawelczyk CLL 144 1 0 27 Sep 2025
A Taxonomy of Transcendence Natalie Abreu Edwin Zhang Eran Malach Sham Kakade 90 2 0 25 Aug 2025
Learning Facts at Scale with Active Reading Jessy Lin Vincent-Pierre Berges Xilun Chen Anuj Kumar Gargi Ghosh Barlas Oğuz RALM KELM 116 2 0 13 Aug 2025
FPEdit: Robust LLM Fingerprinting through Localized Parameter Editing Shida Wang Chaohu Liu Yubo Wang Linli Xu KELM 188 3 0 04 Aug 2025
TARS: MinMax Token-Adaptive Preference Strategy for MLLM Hallucination Reduction Kejia Zhang Keda Tao Zhiming Luo Chang Liu Jiasheng Tang Huan Wang LRM 236 0 0 29 Jul 2025
Rote Learning Considered Useful: Generalizing over Memorized Data in LLMs Qinyuan Wu Soumi Das Mahsa Amani Bishwamittra Ghosh Mohammad Aflah Khan Krishna P. Gummadi Muhammad Bilal Zafar 91 1 0 29 Jul 2025
TokenSmith: Streamlining Data Editing, Search, and Inspection for Large-Scale Language Model Training and Interpretability Mohammad Aflah Khan Ameya Godbole Johnny Tian-Zheng Wei Ryan Yixiang Wang James Flemings Krishna P. Gummadi Willie Neiswanger Robin Jia SyDa 141 0 0 25 Jul 2025
Memorization Sinks: Isolating Memorization during LLM Training Gaurav R. Ghosal Pratyush Maini Aditi Raghunathan MU 203 4 0 14 Jul 2025
Steering Information Utility in Key-Value Memory for Language Model Post-Training Chunyuan Deng Ruidi Chang Hanjie Chen LLMSV 278 0 0 07 Jul 2025
PropMEND: Hypernetworks for Knowledge Propagation in LLMs Zeyu Leo Liu Greg Durrett Eunsol Choi KELM 109 0 0 10 Jun 2025
Multidimensional Analysis of Specific Language Impairment Using Unsupervised Learning Through PCA and ClusteringIEEE International Conference on Healthcare Informatics (ICHI), 2025 Niruthiha Selvanayagam 148 0 0 05 Jun 2025
Bridging Neural ODE and ResNet: A Formal Error Bound for Safety Verification Abdelrahman Sayed Sayed Pierre-Jean Meyer Mohamed Ghazel 163 2 0 03 Jun 2025
Cross-Lingual Transfer of Cultural Knowledge: An Asymmetric PhenomenonAnnual Meeting of the Association for Computational Linguistics (ACL), 2025 Chen Zhang Zhiyuan Liao Yansong Feng 117 2 0 02 Jun 2025
Evaluating the Unseen Capabilities: How Many Theorems Do LLMs Know? Xiang Li Jiayi Xin Qi Long Weijie J. Su ELM 124 1 0 01 Jun 2025
Characterizing Pattern Matching and Its Limits on Compositional Task Structures Hoyeon Chang Jinho Park Hanseul Cho Sohee Yang Miyoung Ko Hyeonbin Hwang Seungpil Won Dohaeng Lee Youbin Ahn Minjoon Seo 214 1 0 26 May 2025
Editing as Unlearning: Are Knowledge Editing Methods Strong Baselines for Large Language Model Unlearning? Zexi Li Xiangzhu Wang William F. Shen Meghdad Kurmanji Xinchi Qiu Dongqi Cai Chao Wu Nicholas D. Lane KELM MU 134 3 0 26 May 2025
Data Mixing Can Induce Phase Transitions in Knowledge Acquisition Xinran Gu Kaifeng Lyu Jiazheng Li Jingzhao Zhang 230 1 0 23 May 2025
Data Whisperer: Efficient Data Selection for Task-Specific LLM Fine-Tuning via Few-Shot In-Context LearningAnnual Meeting of the Association for Computational Linguistics (ACL), 2025 Shaobo Wang Xiangqi Jin Ziming Wang Jinqiao Wang Jingyun Zhang ... Zichen Wen Zhong Li Bin Wang Xuming Hu Linfeng Zhang SyDa 374 13 0 18 May 2025
From n-gram to Attention: How Model Architectures Learn and Propagate Bias in Language Modeling Mohsinul Kabir Tasfia Tahsin Sophia Ananiadou KELM AI4CE 270 1 0 18 May 2025
Bidirectional LMs are Better Knowledge Memorizers? A Benchmark for Real-world Knowledge Injection Yuwei Zhang Wenhao Yu Shangbin Feng Yifan Zhu Letian Peng Jayanth Srinivasa Gaowen Liu Jingbo Shang KELM 224 4 0 18 May 2025
On Linear Representations and Pretraining Data Frequency in Language ModelsInternational Conference on Learning Representations (ICLR), 2025 Jack Merullo Noah A. Smith Sarah Wiegreffe Yanai Elazar 374 10 0 16 Apr 2025
Can Pre-training Indicators Reliably Predict Fine-tuning Outcomes of LLMs? Hansi Zeng Kai Hui Honglei Zhuang Zhen Qin Zhenrui Yue Hamed Zamani Dana Alon 137 0 0 16 Apr 2025
Is the Reversal Curse a Binding Problem? Uncovering Limitations of Transformers from a Basic Generalization Failure Boshi Wang Huan Sun 206 9 0 02 Apr 2025
How do language models learn facts? Dynamics, curricula and hallucinations Nicolas Zucchet J. Bornschein Stephanie C. Y. Chan Andrew Kyle Lampinen Razvan Pascanu Soham De KELM HILM LRM 287 18 1 27 Mar 2025
Evaluating book summaries from internal knowledge in Large Language Models: a cross-model and semantic consistency approach Javier Coronado-Blázquez HILM ELM 207 0 0 27 Mar 2025
Mitigating Memorization in LLMs using Activation Steering Manan Suri Nishit Anand Amisha Bhaskar LLMSV 254 6 0 08 Mar 2025
Robust Data Watermarking in Language Models by Injecting Fictitious KnowledgeAnnual Meeting of the Association for Computational Linguistics (ACL), 2025 Xinyue Cui Johnny Tian-Zheng Wei Swabha Swayamdipta Robin Jia WaLM 496 6 0 06 Mar 2025
SAKE: Steering Activations for Knowledge EditingAnnual Meeting of the Association for Computational Linguistics (ACL), 2025 Marco Scialanga Thibault Laugel Vincent Grari Marcin Detyniecki KELM LLMSV 254 3 0 03 Mar 2025
Interrogating LLM design under a fair learning doctrine Johnny Tian-Zheng Wei Maggie Wang Ameya Godbole Jonathan H. Choi Robin Jia 258 0 0 22 Feb 2025
How Do LLMs Acquire New Knowledge? A Knowledge Circuits Perspective on Continual Pre-TrainingAnnual Meeting of the Association for Computational Linguistics (ACL), 2025 Yixin Ou Yunzhi Yao Ningyu Zhang Hui Jin Jiacheng Sun Shumin Deng Hao Sun Ningyu Zhang KELM CLL 256 9 0 16 Feb 2025
Training an LLM-as-a-Judge Model: Pipeline, Insights, and Practical LessonsThe Web Conference (WWW), 2025 Renjun Hu Yi Cheng Libin Meng Jiaxin Xia Yi Zong Xing Shi Jialin Li 506 11 0 05 Feb 2025
Episodic memory in AI agents poses risks that should be studied and mitigated Chad DeChant 317 5 0 20 Jan 2025
Do Large Language Models Perform Latent Multi-Hop Reasoning without Exploiting Shortcuts?Annual Meeting of the Association for Computational Linguistics (ACL), 2024 Sohee Yang Nora Kassner E. Gribovskaya Sebastian Riedel Mor Geva LRM KELM ReLM 335 16 0 25 Nov 2024
Enhancing LLMs for Power System Simulations: A Feedback-driven Multi-agent FrameworkIEEE Transactions on Smart Grid (IEEE Trans. Smart Grid), 2024 Mengshuo Jia Zeyu Cui Gabriela Hug 307 11 0 21 Nov 2024
Continual Memorization of Factoids in Language Models Howard Chen Jiayi Geng Adithya Bhaskar Dan Friedman Danqi Chen KELM 242 1 0 11 Nov 2024
Scalable Influence and Fact Tracing for Large Language Model PretrainingInternational Conference on Learning Representations (ICLR), 2024 Tyler A. Chang Dheeraj Rajagopal Tolga Bolukbasi Lucas Dixon Ian Tenney TDI 222 15 0 22 Oct 2024
Exploring Forgetting in Large Language Model Pre-TrainingAnnual Meeting of the Association for Computational Linguistics (ACL), 2024 Chonghua Liao Ruobing Xie Xingwu Sun Haowen Sun Zhanhui Kang CLL 182 4 0 22 Oct 2024
ChroKnowledge: Unveiling Chronological Knowledge of Language Models in Multiple DomainsInternational Conference on Learning Representations (ICLR), 2024 Yein Park Chanwoong Yoon Jungwoo Park Donghyeon Lee Minbyul Jeong Jaewoo Kang KELM 349 3 0 13 Oct 2024
How Much Can We Forget about Data Contamination? Sebastian Bordt Suraj Srinivas Valentyn Boreiko U. V. Luxburg 362 8 0 04 Oct 2024
Knowledge Entropy Decay during Language Model Pretraining Hinders New Knowledge AcquisitionInternational Conference on Learning Representations (ICLR), 2024 Jiyeon Kim Hyunji Lee Hyowon Cho Joel Jang Hyeonbin Hwang Seungpil Won Youbin Ahn Dohaeng Lee Minjoon Seo KELM 877 13 0 02 Oct 2024
Diversify and Conquer: Diversity-Centric Data Selection with Iterative Refinement Simon Yu Liangyu Chen Sara Ahmadian Marzieh Fadaee 163 11 0 17 Sep 2024