Don't Make Your LLM an Evaluation Benchmark Cheater

3 November 2023

Yankai Lin

Papers citing "Don't Make Your LLM an Evaluation Benchmark Cheater"

20 / 20 papers shown

Title
LLM-Evaluation Tropes: Perspectives on the Validity of LLM-Evaluations Laura Dietz Oleg Zendel P. Bailey Charles L. A. Clarke Ellese Cotterill Jeff Dalton Faegheh Hasibi Mark Sanderson Nick Craswell ELM 38 0 0 27 Apr 2025
Large language models could be rote learners Yuyang Xu Renjun Hu Haochao Ying J. Wu Xing Shi Wei Lin ELM 41 0 0 11 Apr 2025
Take Caution in Using LLMs as Human Surrogates: Scylla Ex Machina Yuan Gao Dokyun Lee Gordon Burtch Sina Fazelpour LRM 36 7 0 25 Oct 2024
Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptions Yujuan Fu Özlem Uzuner Meliha Yetisgen Fei Xia 31 3 0 24 Oct 2024
NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples Baiqi Li Zhiqiu Lin Wenxuan Peng Jean de Dieu Nyandwi Daniel Jiang Zixian Ma Simran Khanuja Ranjay Krishna Graham Neubig Deva Ramanan AAML CoGe VLM 39 20 0 18 Oct 2024
Detecting Training Data of Large Language Models via Expectation Maximization Gyuwan Kim Yang Li Evangelia Spiliopoulou Jie Ma Miguel Ballesteros William Yang Wang MIALM 80 3 2 10 Oct 2024
Fine-tuning can Help Detect Pretraining Data from Large Language Models H. Zhang Songxin Zhang Bingyi Jing Hongxin Wei 31 0 0 09 Oct 2024
Training on the Benchmark Is Not All You Need Shiwen Ni Xiangtao Kong Chengming Li Xiping Hu Ruifeng Xu Jia Zhu Min Yang 34 5 0 03 Sep 2024
Bringing AI Participation Down to Scale: A Comment on Open AIs Democratic Inputs to AI Project David Moats Chandrima Ganguly VLM 27 0 0 16 Jul 2024
Training on the Test Task Confounds Evaluation and Emergence Ricardo Dominguez-Olmedo Florian E. Dorner Moritz Hardt ELM 42 6 1 10 Jul 2024
Benchmark Data Contamination of Large Language Models: A Survey Cheng Xu Shuhao Guan Derek Greene Mohand-Tahar Kechadi ELM ALM 24 38 0 06 Jun 2024
Exploring Subjectivity for more Human-Centric Assessment of Social Biases in Large Language Models Paula Akemi Aoyagui Sharon Ferguson Anastasia Kuzminykh 23 0 0 17 May 2024
Examining the robustness of LLM evaluation to the distributional assumptions of benchmarks Melissa Ailem Katerina Marazopoulou Charlotte Siska James Bono 49 13 0 25 Apr 2024
Evaluating the Factuality of Large Language Models using Large-Scale Knowledge Graphs Xiaoze Liu Feijie Wu Tianyang Xu Zhuo Chen Yichi Zhang Xiaoqian Wang Jing Gao HILM 27 8 0 01 Apr 2024
Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance Jiasheng Ye Peiju Liu Tianxiang Sun Yunhua Zhou Jun Zhan Xipeng Qiu 29 58 0 25 Mar 2024
Spurious Feature Eraser: Stabilizing Test-Time Adaptation for Vision-Language Foundation Model Huan Ma Yan Zhu Changqing Zhang Peilin Zhao Baoyuan Wu Long-Kai Huang Qinghua Hu Bing Wu VLM 39 1 0 01 Mar 2024
Institutional Platform for Secure Self-Service Large Language Model Exploration V. Bumgardner Mitchell A. Klusty W. V. Logan Samuel E. Armstrong Caylin D. Hickey Jeff Talbert Caylin Hickey Jeff Talbert 30 1 0 01 Feb 2024
Fake Alignment: Are LLMs Really Aligned Well? Yixu Wang Yan Teng Kexin Huang Chengqi Lyu Songyang Zhang Wenwei Zhang Xingjun Ma Yu-Gang Jiang Yu Qiao Yingchun Wang 12 14 0 10 Nov 2023
Can we trust the evaluation on ChatGPT? Rachith Aiyappa Jisun An Haewoon Kwak Yong-Yeol Ahn ELM ALM LLMAG AI4MH LRM 98 76 0 22 Mar 2023
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 236 1,508 0 31 Dec 2020