How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering

2 December 2020

Graham Neubig

Papers citing "How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering"

50 / 73 papers shown

Title
Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach Jiancong Xiao Bojian Hou Zhanliang Wang Ruochen Jin Q. Long Weijie Su Li Shen 30 0 0 04 May 2025
What do Language Model Probabilities Represent? From Distribution Estimation to Response Prediction Eitan Wagner Omri Abend 36 0 0 04 May 2025
Always Tell Me The Odds: Fine-grained Conditional Probability Estimation Liaoyaqi Wang Zhengping Jiang Anqi Liu Benjamin Van Durme 61 0 0 02 May 2025
Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Review Toghrul Abbasli Kentaroh Toyoda Yuan Wang Leon Witt Muhammad Asif Ali Yukai Miao Dan Li Qingsong Wei UQCV 92 0 0 25 Apr 2025
A Survey on Knowledge-Oriented Retrieval-Augmented Generation Mingyue Cheng Yucong Luo Jie Ouyang Q. Liu Huijie Liu ... Bohou Zhang Jiawei Cao Jie Ma Daoyu Wang Enhong Chen 3DV 70 3 0 11 Mar 2025
Societal Alignment Frameworks Can Improve LLM Alignment Karolina Stañczak Nicholas Meade Mehar Bhatia Hattie Zhou Konstantin Böttinger ... Timothy P. Lillicrap Ana Marasović Sylvie Delacroix Gillian K. Hadfield Siva Reddy 144 0 0 27 Feb 2025
Large Language Model Confidence Estimation via Black-Box Access Tejaswini Pedapati Amit Dhurandhar Soumya Ghosh Soham Dan P. Sattigeri 89 3 0 21 Feb 2025
Mind the Confidence Gap: Overconfidence, Calibration, and Distractor Effects in Large Language Models Prateek Chhikara 39 1 0 16 Feb 2025
Unveiling the Capabilities of Large Language Models in Detecting Offensive Language with Annotation Disagreement Junyu Lu Kai Ma Kaichun Wang Kelaiti Xiao Roy Ka-Wei Lee Bo Xu Liang Yang Hongfei Lin 49 0 0 10 Feb 2025
Confidence Elicitation: A New Attack Vector for Large Language Models Brian Formento Chuan-Sheng Foo See-Kiong Ng AAML 99 0 0 07 Feb 2025
A statistically consistent measure of Semantic Variability using Language Models Yi Liu 71 0 0 01 Feb 2025
Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologies L. Wang Sheng Chen Linnan Jiang Shu Pan Runze Cai Sen Yang Fei Yang 49 3 0 24 Oct 2024
Fact Recall, Heuristics or Pure Guesswork? Precise Interpretations of Language Models for Fact Completion Denitsa Saynova Lovisa Hagström Moa Johansson Richard Johansson Marco Kuhlmann HILM 43 0 0 18 Oct 2024
On Calibration of LLM-based Guard Models for Reliable Content Moderation Hongfu Liu Hengguan Huang Hao Wang Xiangming Gu Ye Wang 55 2 0 14 Oct 2024
Calibrating Expressions of Certainty Peiqi Wang Barbara D. Lam Yingcheng Liu Ameneh Asgari-Targhi Rameswar Panda W. Wells Tina Kapur Polina Golland 32 1 0 06 Oct 2024
Integrative Decoding: Improve Factuality via Implicit Self-consistency Yi Cheng Xiao Liang Yeyun Gong Wen Xiao Song Wang ... Wenjie Li Jian Jiao Qi Chen Peng Cheng Wayne Xiong HILM 56 1 0 02 Oct 2024
Knowledge Planning in Large Language Models for Domain-Aligned Counseling Summarization Aseem Srivastava Smriti Joshi Tanmoy Chakraborty Md. Shad Akhtar 32 3 0 23 Sep 2024
Zero-resource Hallucination Detection for Text Generation via Graph-based Contextual Knowledge Triples Modeling Xinyue Fang Zhen Huang Zhiliang Tian Minghui Fang Ziyi Pan Quntian Fang Zhihua Wen Hengyue Pan Dongsheng Li HILM 91 2 0 17 Sep 2024
Teaching LLMs to Abstain across Languages via Multilingual Feedback Shangbin Feng Weijia Shi Yike Wang Wenxuan Ding Orevaoghene Ahia Shuyue Stella Li Vidhisha Balachandran Sunayana Sitaram Yulia Tsvetkov 67 4 0 22 Jun 2024
Reassessing How to Compare and Improve the Calibration of Machine Learning Models M. Chidambaram Rong Ge 68 1 0 06 Jun 2024
SUKHSANDESH: An Avatar Therapeutic Question Answering Platform for Sexual Education in Rural India Salam Michael Singh Shubhmoy Kumar Garg Amitesh Misra Aaditeshwar Seth Tanmoy Chakraborty 31 0 0 03 May 2024
Language Model Cascades: Token-level uncertainty and beyond Neha Gupta Harikrishna Narasimhan Wittawat Jitkrittum A. S. Rawat A. Menon Sanjiv Kumar UQLM 47 42 0 15 Apr 2024
LoRA Dropout as a Sparsity Regularizer for Overfitting Control Yang Lin Xinyu Ma Xu Chu Yujie Jin Zhibang Yang Yasha Wang Hong-yan Mei 49 19 0 15 Apr 2024
Confidence Calibration and Rationalization for LLMs via Multi-Agent Deliberation Ruixin Yang Dheeraj Rajagopal S. Hayati Bin Hu Dongyeop Kang LLMAG 40 3 0 14 Apr 2024
Calibrating the Confidence of Large Language Models by Eliciting Fidelity Mozhi Zhang Mianqiu Huang Rundong Shi Linsen Guo Chong Peng Peng Yan Yaqian Zhou Xipeng Qiu 22 10 0 03 Apr 2024
Learning to Maximize Mutual Information for Chain-of-Thought Distillation Xin Chen Hanxian Huang Yanjun Gao Yi Wang Jishen Zhao Ke Ding 35 11 0 05 Mar 2024
Calibrating Large Language Models with Sample Consistency Qing Lyu Kumar Shridhar Chaitanya Malaviya Li Zhang Yanai Elazar Niket Tandon Marianna Apidianaki Mrinmaya Sachan Chris Callison-Burch 43 23 0 21 Feb 2024
Introspective Planning: Aligning Robots' Uncertainty with Inherent Task Ambiguity Kaiqu Liang Zixu Zhang J. F. Fisac LLMAG 46 6 0 09 Feb 2024
Distinguishing the Knowable from the Unknowable with Language Models Gustaf Ahdritz Tian Qin Nikhil Vyas Boaz Barak Benjamin L. Edelman 24 18 0 05 Feb 2024
Learning Shortcuts: On the Misleading Promise of NLU in Language Models Geetanjali Bihani Julia Taylor Rayz 33 3 0 17 Jan 2024
Robust Knowledge Extraction from Large Language Models using Social Choice Theory Nico Potyka Yuqicheng Zhu Yunjie He Evgeny Kharlamov Steffen Staab 24 3 0 22 Dec 2023
Calibrated Language Models Must Hallucinate Adam Tauman Kalai Santosh Vempala HILM 22 75 0 24 Nov 2023
Probabilistic Tree-of-thought Reasoning for Answering Knowledge-intensive Complex Questions S. Cao Jiajie Zhang Jiaxin Shi Xin Lv Zijun Yao Qingwen Tian Juanzi Li Lei Hou LRM 29 13 0 23 Nov 2023
Knowing What LLMs DO NOT Know: A Simple Yet Effective Self-Detection Method Yukun Zhao Lingyong Yan Weiwei Sun Guoliang Xing Chong Meng Shuaiqiang Wang Zhicong Cheng Zhaochun Ren Dawei Yin 27 35 0 27 Oct 2023
Merging Generated and Retrieved Knowledge for Open-Domain QA Yunxiang Zhang Muhammad Khalifa Lajanugen Logeswaran Moontae Lee Honglak Lee Lu Wang RALM 28 37 0 22 Oct 2023
Chain-of-Thought Tuning: Masked Language Models can also Think Step By Step in Natural Language Understanding Caoyun Fan Jidong Tian Yitian Li Wenqing Chen Hao He Yaohui Jin LRM 32 3 0 18 Oct 2023
Cognitive Architectures for Language Agents T. Sumers Shunyu Yao Karthik Narasimhan Thomas L. Griffiths LLMAG LM&Ro 45 152 0 05 Sep 2023
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models Yue Zhang Yafu Li Leyang Cui Deng Cai Lemao Liu ... Longyue Wang A. Luu Wei Bi Freda Shi Shuming Shi RALM LRM HILM 43 519 0 03 Sep 2023
Thrust: Adaptively Propels Large Language Models with External Knowledge Xinran Zhao Hongming Zhang Xiaoman Pan Wenlin Yao Dong Yu Jianshu Chen KELM 48 4 0 19 Jul 2023
Unsupervised Calibration through Prior Adaptation for Text Classification using Large Language Models Lautaro Estienne Luciana Ferrer Matías Vera Pablo Piantanida VLM 26 1 0 13 Jul 2023
Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners Allen Z. Ren Anushri Dixit Alexandra Bodrova Sumeet Singh Stephen Tu ... Jacob Varley Zhenjia Xu Dorsa Sadigh Andy Zeng Anirudha Majumdar LM&Ro 49 219 0 04 Jul 2023
Personality Traits in Large Language Models Gregory Serapio-García Mustafa Safdari Clément Crepy Luning Sun Stephen Fitz P. Romero Marwa Abdulhai Aleksandra Faust Maja J. Matarić LM&MA LLMAG 58 119 0 01 Jul 2023
AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap Q. V. Liao J. Vaughan 36 158 0 02 Jun 2023
Estimating Large Language Model Capabilities without Labeled Test Data Harvey Yiyun Fu Qinyuan Ye Albert Xu Xiang Ren Robin Jia 21 8 0 24 May 2023
Knowledge of Knowledge: Exploring Known-Unknowns Uncertainty with Large Language Models Alfonso Amayuelas Kyle Wong Liangming Pan Wenhu Chen W. Wang 37 26 0 23 May 2023
Active Retrieval Augmented Generation Zhengbao Jiang Frank F. Xu Luyu Gao Zhiqing Sun Qian Liu Jane Dwivedi-Yu Yiming Yang Jamie Callan Graham Neubig RALM 9 252 0 11 May 2023
Taking Advice from ChatGPT Peter Zhang 34 5 0 11 May 2023
Calibration Error Estimation Using Fuzzy Binning Geetanjali Bihani Julia Taylor Rayz 93 2 0 30 Apr 2023
Context-Aware Differential Privacy for Language Modeling M. H. Dinh Ferdinando Fioretto 25 2 0 28 Jan 2023
SPE: Symmetrical Prompt Enhancement for Fact Probing Yiyuan Li Tong Che Yezhen Wang Zhengbao Jiang Caiming Xiong Snigdha Chaturvedi 26 6 0 14 Nov 2022