Large Language Models Struggle to Learn Long-Tail Knowledge

15 November 2022

Papers citing "Large Language Models Struggle to Learn Long-Tail Knowledge"

48 / 248 papers shown

Title
Can Language Models be Instructed to Protect Personal Information? Yang Chen Ethan Mendes Sauvik Das Wei-ping Xu Alan Ritter PILM 11 34 0 03 Oct 2023
Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of Language Models Mert Yuksekgonul Varun Chandrasekaran Erik Jones Suriya Gunasekar Ranjita Naik Hamid Palangi Ece Kamar Besmira Nushi HILM 13 39 0 26 Sep 2023
Ragas: Automated Evaluation of Retrieval Augmented Generation ES Shahul Jithin James Luis Espinosa-Anke Steven Schockaert 80 174 0 26 Sep 2023
The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A" Lukas Berglund Meg Tong Max Kaufmann Mikita Balesni Asa Cooper Stickland Tomasz Korbak Owain Evans LRM 20 233 0 21 Sep 2023
Cognitive Mirage: A Review of Hallucinations in Large Language Models Hongbin Ye Tong Liu Aijia Zhang Wei Hua Weiqiang Jia HILM 37 76 0 13 Sep 2023
Loquacity and Visible Emotion: ChatGPT as a Policy Advisor Claudia Biancotti C. Camassa 16 2 0 07 Sep 2023
Explainability for Large Language Models: A Survey Haiyan Zhao Hanjie Chen Fan Yang Ninghao Liu Huiqi Deng Hengyi Cai Shuaiqiang Wang Dawei Yin Mengnan Du LRM 19 404 0 02 Sep 2023
Characterizing Learning Curves During Language Model Pre-Training: Learning, Forgetting, and Stability Tyler A. Chang Z. Tu Benjamin Bergen 16 10 0 29 Aug 2023
Journey to the Center of the Knowledge Neurons: Discoveries of Language-Independent Knowledge Neurons and Degenerate Knowledge Neurons Yuheng Chen Pengfei Cao Yubo Chen Kang Liu Jun Zhao KELM 23 41 0 25 Aug 2023
Head-to-Tail: How Knowledgeable are Large Language Models (LLMs)? A.K.A. Will LLMs Replace Knowledge Graphs? Kai Sun Y. Xu Hanwen Zha Yue Liu Xinhsuai Dong AI4MH 25 130 0 20 Aug 2023
KnowledGPT: Enhancing Large Language Models with Retrieval and Storage Access on Knowledge Bases Xintao Wang Qian Yang Yongting Qiu Jiaqing Liang Qi He Zhouhong Gu Yanghua Xiao W. Wang KELM RALM 34 39 0 17 Aug 2023
LLaMA-E: Empowering E-commerce Authoring with Object-Interleaved Instruction Following Kaize Shi Xueyao Sun Dingxian Wang Yinlin Fu Guandong Xu Qing Li 15 3 0 09 Aug 2023
The Bias Amplification Paradox in Text-to-Image Generation P. Seshadri Sameer Singh Yanai Elazar DiffM 11 39 0 01 Aug 2023
Lost in the Middle: How Language Models Use Long Contexts Nelson F. Liu Kevin Lin John Hewitt Ashwin Paranjape Michele Bevilacqua Fabio Petroni Percy Liang RALM 27 1,380 0 06 Jul 2023
Offline Reinforcement Learning with Imbalanced Datasets Li Jiang Sijie Cheng Jielin Qiu Haoran Xu Wai Kin Victor Chan Zhao Ding OffRL 16 3 0 06 Jul 2023
Bring Your Own Data! Self-Supervised Evaluation for Large Language Models Neel Jain Khalid Saifullah Yuxin Wen John Kirchenbauer Manli Shu Aniruddha Saha Micah Goldblum Jonas Geiping Tom Goldstein ALM ELM 19 22 0 23 Jun 2023
Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modeling Lin F. Yang Hongyang Chen Zhao Li Xiao Ding Xindong Wu KELM 15 83 0 20 Jun 2023
Large Language Models, scientific knowledge and factuality: A systematic analysis in antibiotic discovery Magdalena Wysocka Oskar Wysocki Maxime Delmas V. Mutel André Freitas LM&MA 25 6 0 28 May 2023
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation Sewon Min Kalpesh Krishna Xinxi Lyu M. Lewis Wen-tau Yih Pang Wei Koh Mohit Iyyer Luke Zettlemoyer Hannaneh Hajishirzi HILM ALM 14 595 0 23 May 2023
Improving Language Models via Plug-and-Play Retrieval Feedback W. Yu Zhihan Zhang Zhenwen Liang Meng-Long Jiang Ashish Sabharwal KELM LRM 22 71 0 23 May 2023
"According to ...": Prompting Language Models Improves Quoting from Pre-Training Data Orion Weller Marc Marone Nathaniel Weir Dawn J Lawrie Daniel Khashabi Benjamin Van Durme HILM 66 44 0 22 May 2023
Quantifying Association Capabilities of Large Language Models and Its Implications on Privacy Leakage Hanyin Shao Jie Huang Shen Zheng Kevin Chen-Chuan Chang PILM 8 24 0 22 May 2023
Knowledge Card: Filling LLMs' Knowledge Gaps with Plug-in Specialized Language Models Shangbin Feng Weijia Shi Yuyang Bai Vidhisha Balachandran Tianxing He Yulia Tsvetkov KELM 45 28 0 17 May 2023
Can LMs Learn New Entities from Descriptions? Challenges in Propagating Injected Knowledge Yasumasa Onoe Michael J.Q. Zhang Shankar Padmanabhan Greg Durrett Eunsol Choi KELM 201 73 0 02 May 2023
Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4 Kent K. Chang Mackenzie Cramer Sandeep Soni David Bamman RALM 138 109 0 28 Apr 2023
Search-in-the-Chain: Interactively Enhancing Large Language Models with Search for Knowledge-intensive Tasks Shicheng Xu Liang Pang Huawei Shen Xueqi Cheng Tat-Seng Chua RALM KELM LRM 27 37 0 28 Apr 2023
The MiniPile Challenge for Data-Efficient Language Models Jean Kaddour MoE ALM 10 41 0 17 Apr 2023
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling Stella Biderman Hailey Schoelkopf Quentin G. Anthony Herbie Bradley Kyle O'Brien ... USVSN Sai Prashanth Edward Raff Aviya Skowron Lintang Sutawika Oskar van der Wal 25 1,160 0 03 Apr 2023
Recognition, recall, and retention of few-shot memories in large language models A. Orhan LRM KELM CLL 27 3 0 30 Mar 2023
Language Model Behavior: A Comprehensive Survey Tyler A. Chang Benjamin Bergen VLM LRM LM&MA 27 100 0 20 Mar 2023
Secret-Keeping in Question Answering Nathaniel W. Rollings Kent O'Sullivan Sakshum Kulshrestha KELM 16 0 0 16 Mar 2023
Testing AI on language comprehension tasks reveals insensitivity to underlying meaning Vittoria Dentella Fritz Guenther Elliot Murphy G. Marcus Evelina Leivada ELM 25 24 0 23 Feb 2023
Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions? Yang Chen Hexiang Hu Yi Luan Haitian Sun Soravit Changpinyo Alan Ritter Ming-Wei Chang 21 79 0 23 Feb 2023
Conversational AI-Powered Design: ChatGPT as Designer, User, and Product A. Kocaballi 19 38 0 15 Feb 2023
Extracting Training Data from Diffusion Models Nicholas Carlini Jamie Hayes Milad Nasr Matthew Jagielski Vikash Sehwag Florian Tramèr Borja Balle Daphne Ippolito Eric Wallace DiffM 23 568 0 30 Jan 2023
Self-Instruct: Aligning Language Models with Self-Generated Instructions Yizhong Wang Yeganeh Kordi Swaroop Mishra Alisa Liu Noah A. Smith Daniel Khashabi Hannaneh Hajishirzi ALM SyDa LRM 17 2,042 0 20 Dec 2022
A Survey of Deep Learning for Mathematical Reasoning Pan Lu Liang Qiu Wenhao Yu Sean Welleck Kai-Wei Chang ReLM LRM 32 137 0 20 Dec 2022
When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories Alex Troy Mallen Akari Asai Victor Zhong Rajarshi Das Daniel Khashabi Hannaneh Hajishirzi RALM HILM KELM 6 505 0 20 Dec 2022
Demystifying Prompts in Language Models via Perplexity Estimation Hila Gonen Srini Iyer Terra Blevins Noah A. Smith Luke Zettlemoyer LRM 25 194 0 08 Dec 2022
Nonparametric Masked Language Modeling Sewon Min Weijia Shi M. Lewis Xilun Chen Wen-tau Yih Hannaneh Hajishirzi Luke Zettlemoyer RALM 40 48 0 02 Dec 2022
Understanding BLOOM: An empirical study on diverse NLP tasks Parag Dakle Sai Krishna Rallabandi Preethi Raghavan AI4CE 15 3 0 27 Nov 2022
Benchmarking Long-tail Generalization with Likelihood Splits Ameya Godbole Robin Jia ALM 13 8 0 13 Oct 2022
Frequency Effects on Syntactic Rule Learning in Transformers Jason W. Wei Dan Garrette Tal Linzen Ellie Pavlick 80 62 0 14 Sep 2021
Deduplicating Training Data Makes Language Models Better Katherine Lee Daphne Ippolito A. Nystrom Chiyuan Zhang Douglas Eck Chris Callison-Burch Nicholas Carlini SyDa 237 588 0 14 Jul 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 245 1,977 0 31 Dec 2020
Extracting Training Data from Large Language Models Nicholas Carlini Florian Tramèr Eric Wallace Matthew Jagielski Ariel Herbert-Voss ... Tom B. Brown D. Song Ulfar Erlingsson Alina Oprea Colin Raffel MLAU SILM 267 1,798 0 14 Dec 2020
Distilling Knowledge from Reader to Retriever for Question Answering Gautier Izacard Edouard Grave RALM 176 249 0 08 Dec 2020
Language Models as Knowledge Bases? Fabio Petroni Tim Rocktaschel Patrick Lewis A. Bakhtin Yuxiang Wu Alexander H. Miller Sebastian Riedel KELM AI4MH 398 2,576 0 03 Sep 2019