How Do Large Language Models Acquire Factual Knowledge During Pretraining?

17 June 2024

Du-Seong Chang

Minjoon Seo

KELM

ArXiv PDF HTML

Papers citing "How Do Large Language Models Acquire Factual Knowledge During Pretraining?"

31 / 31 papers shown

Title
On Linear Representations and Pretraining Data Frequency in Language Models Jack Merullo Noah A. Smith Sarah Wiegreffe Yanai Elazar 30 0 0 16 Apr 2025
Can Pre-training Indicators Reliably Predict Fine-tuning Outcomes of LLMs? Hansi Zeng Kai Hui Honglei Zhuang Zhen Qin Zhenrui Yue Hamed Zamani Dana Alon 33 0 0 16 Apr 2025
Is the Reversal Curse a Binding Problem? Uncovering Limitations of Transformers from a Basic Generalization Failure Boshi Wang Huan Sun 31 2 0 02 Apr 2025
How do language models learn facts? Dynamics, curricula and hallucinations Nicolas Zucchet J. Bornschein Stephanie C. Y. Chan Andrew Kyle Lampinen Razvan Pascanu Soham De KELM HILM LRM 65 1 1 27 Mar 2025
Evaluating book summaries from internal knowledge in Large Language Models: a cross-model and semantic consistency approach Javier Coronado-Blázquez HILM ELM 59 0 0 27 Mar 2025
Mitigating Memorization in LLMs using Activation Steering Manan Suri Nishit Anand Amisha Bhaskar LLMSV 45 2 0 08 Mar 2025
Robust Data Watermarking in Language Models by Injecting Fictitious Knowledge Xinyue Cui Johnny Tian-Zheng Wei Swabha Swayamdipta Robin Jia WaLM 76 0 0 06 Mar 2025
SAKE: Steering Activations for Knowledge Editing Marco Scialanga Thibault Laugel Vincent Grari Marcin Detyniecki KELM LLMSV 60 1 0 03 Mar 2025
Interrogating LLM design under a fair learning doctrine Johnny Tian-Zheng Wei Maggie Wang Ameya Godbole Jonathan H. Choi Robin Jia 24 0 0 22 Feb 2025
How Do LLMs Acquire New Knowledge? A Knowledge Circuits Perspective on Continual Pre-Training Yixin Ou Yunzhi Yao N. Zhang Hui Jin Jiacheng Sun Shumin Deng Z. Li H. Chen KELM CLL 49 0 0 16 Feb 2025
Episodic memory in AI agents poses risks that should be studied and mitigated Chad DeChant 53 1 0 20 Jan 2025
Do Large Language Models Perform Latent Multi-Hop Reasoning without Exploiting Shortcuts? Sohee Yang Nora Kassner E. Gribovskaya Sebastian Riedel Mor Geva KELM LRM ReLM 70 4 0 25 Nov 2024
Enhancing LLMs for Power System Simulations: A Feedback-driven Multi-agent Framework Mengshuo Jia Zeyu Cui Gabriela Hug 67 0 0 21 Nov 2024
Continual Memorization of Factoids in Language Models Howard Chen Jiayi Geng Adithya Bhaskar Dan Friedman Danqi Chen KELM 29 0 0 11 Nov 2024
Scalable Influence and Fact Tracing for Large Language Model Pretraining Tyler A. Chang Dheeraj Rajagopal Tolga Bolukbasi Lucas Dixon Ian Tenney TDI 23 0 0 22 Oct 2024
Exploring Forgetting in Large Language Model Pre-Training Chonghua Liao Ruobing Xie X. Sun Haowen Sun Zhanhui Kang CLL 25 0 0 22 Oct 2024
ChroKnowledge: Unveiling Chronological Knowledge of Language Models in Multiple Domains Yein Park Chanwoong Yoon Jungwoo Park Donghyeon Lee Minbyul Jeong Jaewoo Kang KELM 38 1 0 13 Oct 2024
How Much Can We Forget about Data Contamination? Sebastian Bordt Suraj Srinivas Valentyn Boreiko U. V. Luxburg 36 1 0 04 Oct 2024
Knowledge Entropy Decay during Language Model Pretraining Hinders New Knowledge Acquisition Jiyeon Kim Hyunji Lee Hyowon Cho Joel Jang Hyeonbin Hwang Seungpil Won Youbin Ahn Dohaeng Lee Minjoon Seo KELM 41 2 0 02 Oct 2024
Diversify and Conquer: Diversity-Centric Data Selection with Iterative Refinement Simon Yu Liangyu Chen Sara Ahmadian Marzieh Fadaee 29 6 0 17 Sep 2024
Beneath the Surface of Consistency: Exploring Cross-lingual Knowledge Representation Sharing in LLMs Maxim Ifergan Leshem Choshen Roee Aharoni Idan Szpektor Omri Abend HILM 30 3 0 20 Aug 2024
Demystifying Verbatim Memorization in Large Language Models Jing Huang Diyi Yang Christopher Potts ELM PILM MU 32 1 0 25 Jul 2024
AI-native Memory: A Pathway from LLMs Towards AGI Jingbo Shang Zai Zheng Jiale Wei Xiang Ying Felix Tao Mindverse Team LLMAG 34 7 0 26 Jun 2024
Evaluating $n$ -Gram Novelty of Language Models Using Rusty-DAWG William Merrill Noah A. Smith Yanai Elazar ELM TDI 32 9 0 18 Jun 2024
OLMo: Accelerating the Science of Language Models Dirk Groeneveld Iz Beltagy Pete Walsh Akshita Bhagia Rodney Michael Kinney ... Jesse Dodge Kyle Lo Luca Soldaini Noah A. Smith Hanna Hajishirzi OSLM 124 349 0 01 Feb 2024
To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis Fuzhao Xue Yao Fu Wangchunshu Zhou Zangwei Zheng Yang You 76 74 0 22 May 2023
Deduplicating Training Data Makes Language Models Better Katherine Lee Daphne Ippolito A. Nystrom Chiyuan Zhang Douglas Eck Chris Callison-Burch Nicholas Carlini SyDa 234 447 0 14 Jul 2021
Memorisation versus Generalisation in Pre-trained Language Models Michael Tänzer Sebastian Ruder Marek Rei 81 50 0 16 Apr 2021
Analyzing Commonsense Emergence in Few-shot Knowledge Models Jeff Da Ronan Le Bras Ximing Lu Yejin Choi Antoine Bosselut AI4MH KELM 62 40 0 01 Jan 2021
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 220 3,054 0 23 Jan 2020
Language Models as Knowledge Bases? Fabio Petroni Tim Rocktaschel Patrick Lewis A. Bakhtin Yuxiang Wu Alexander H. Miller Sebastian Riedel KELM AI4MH 393 2,216 0 03 Sep 2019