How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering

2 December 2020

Graham Neubig

Papers citing "How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering"

24 / 74 papers shown

Title
SPE: Symmetrical Prompt Enhancement for Fact Probing Yiyuan Li Tong Che Yezhen Wang Zhengbao Jiang Caiming Xiong Snigdha Chaturvedi 26 6 0 14 Nov 2022
The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation Barbara Plank 30 97 0 04 Nov 2022
Stop Measuring Calibration When Humans Disagree Joris Baan Wilker Aziz Barbara Plank Raquel Fernández 24 53 0 28 Oct 2022
NormSAGE: Multi-Lingual Multi-Cultural Norm Discovery from Conversations On-the-Fly Yi Ren Fung Tuhin Chakraborty Hao Guo Owen Rambow Smaranda Muresan Heng Ji 21 39 0 16 Oct 2022
CHARD: Clinical Health-Aware Reasoning Across Dimensions for Text Generation Models Steven Y. Feng Vivek Khetan Bogdan Sacaleanu A. Gershman Eduard H. Hovy LRM 27 10 0 09 Oct 2022
Selecting Better Samples from Pre-trained LLMs: A Case Study on Question Generation Xingdi Yuan Tong Wang Yen-Hsiang Wang Emery Fine Rania Abdelghani Pauline Lucas Hélene Sauzéon Pierre-Yves Oudeyer 30 29 0 22 Sep 2022
On the Relation between Sensitivity and Accuracy in In-context Learning Yanda Chen Chen Zhao Zhou Yu Kathleen McKeown He He 182 77 0 16 Sep 2022
Meaning without reference in large language models S. Piantadosi Felix Hill 14 72 0 05 Aug 2022
Language models show human-like content effects on reasoning tasks Ishita Dasgupta Andrew Kyle Lampinen Stephanie C. Y. Chan Hannah R. Sheahan Antonia Creswell D. Kumaran James L. McClelland Felix Hill ReLM LRM 28 181 0 14 Jul 2022
Language Models (Mostly) Know What They Know Saurav Kadavath Tom Conerly Amanda Askell T. Henighan Dawn Drain ... Nicholas Joseph Benjamin Mann Sam McCandlish C. Olah Jared Kaplan ELM 47 712 0 11 Jul 2022
Re-Examining Calibration: The Case of Question Answering Chenglei Si Chen Zhao Sewon Min Jordan L. Boyd-Graber 61 30 0 25 May 2022
Learning a Better Initialization for Soft Prompts via Meta-Learning Yukun Huang Kun Qian Zhou Yu VLM 39 9 0 25 May 2022
Life after BERT: What do Other Muppets Understand about Language? Vladislav Lialin Kevin Zhao Namrata Shivagunde Anna Rumshisky 39 6 0 21 May 2022
Calibration of Natural Language Understanding Models with Venn--ABERS Predictors Patrizio Giovannotti 38 6 0 21 May 2022
Decorate the Examples: A Simple Method of Prompt Design for Biomedical Relation Extraction Hui-Syuan Yeh Thomas Lavergne Pierre Zweigenbaum 21 10 0 21 Apr 2022
Uncertainty Estimation for Language Reward Models Adam Gleave G. Irving UQLM 29 31 0 14 Mar 2022
Semantic-Oriented Unlabeled Priming for Large-Scale Language Models Yanchen Liu Timo Schick Hinrich Schütze VLM 28 15 0 12 Feb 2022
What Has Been Enhanced in my Knowledge-Enhanced Language Model? Yifan Hou Guoji Fu Mrinmaya Sachan KELM 33 1 0 02 Feb 2022
Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey Bonan Min Hayley L Ross Elior Sulem Amir Pouran Ben Veyseh Thien Huu Nguyen Oscar Sainz Eneko Agirre Ilana Heinz Dan Roth LM&MA VLM AI4CE 74 1,030 0 01 Nov 2021
Towards Language Modelling in the Speech Domain Using Sub-word Linguistic Units Anurag Katakkar A. Black AuLLM 22 1 0 31 Oct 2021
Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing Pengfei Liu Weizhe Yuan Jinlan Fu Zhengbao Jiang Hiroaki Hayashi Graham Neubig VLM SyDa 31 3,831 0 28 Jul 2021
Calibration of Pre-trained Transformers Shrey Desai Greg Durrett UQLM 243 289 0 17 Mar 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 237 4,469 0 23 Jan 2020
Language Models as Knowledge Bases? Fabio Petroni Tim Rocktaschel Patrick Lewis A. Bakhtin Yuxiang Wu Alexander H. Miller Sebastian Riedel KELM AI4MH 415 2,586 0 03 Sep 2019