Training on the Test Task Confounds Evaluation and Emergence

Training on the Test Task Confounds Evaluation and Emergence

10 July 2024

Ricardo Dominguez-Olmedo

Florian E. Dorner

Papers citing "Training on the Test Task Confounds Evaluation and Emergence"

15 / 15 papers shown

Title
A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility Andreas Hochlehnert Hardik Bhatnagar Vishaal Udandarao Samuel Albanie Ameya Prabhu Matthias Bethge ReLM ALM LRM 58 4 0 09 Apr 2025
YourBench: Easy Custom Evaluation Sets for Everyone S. Kamath S Clémentine Fourrier Alina Lozovskia Thomas Wolf Gökhan Tür Dilek Hakkani-Tür 30 1 0 02 Apr 2025
Leaving the barn door open for Clever Hans: Simple features predict LLM benchmark answers Lorenzo Pacchiardi Marko Tesic Lucy G. Cheke José Hernández Orallo 28 3 0 15 Oct 2024
DataComp-LM: In search of the next generation of training sets for language models Jeffrey Li Alex Fang Georgios Smyrnis Maor Ivgi Matt Jordan ... Alexandros G. Dimakis Y. Carmon Achal Dave Ludwig Schmidt Vaishaal Shankar ELM 27 79 0 17 Jun 2024
Gemma: Open Models Based on Gemini Research and Technology Gemma Team Gemma Team Thomas Mesnard Cassidy Hardin Robert Dadashi Surya Bhupatiraju ... Armand Joulin Noah Fiedel Evan Senter Alek Andreev Kathleen Kenealy VLM LLMAG 120 415 0 13 Mar 2024
OLMo: Accelerating the Science of Language Models Dirk Groeneveld Iz Beltagy Pete Walsh Akshita Bhagia Rodney Michael Kinney ... Jesse Dodge Kyle Lo Luca Soldaini Noah A. Smith Hanna Hajishirzi OSLM 124 128 0 01 Feb 2024
Task Contamination: Language Models May Not Be Few-Shot Anymore Changmao Li Jeffrey Flanigan 65 87 0 26 Dec 2023
The Falcon Series of Open Language Models Ebtesam Almazrouei Hamza Alobeidli Abdulaziz Alshamsi Alessandro Cappelli Ruxandra-Aimée Cojocaru ... Quentin Malartic Daniele Mazzotta Badreddine Noune B. Pannier Guilherme Penedo AI4TS ALM 104 218 0 28 Nov 2023
Don't Make Your LLM an Evaluation Benchmark Cheater Kun Zhou Yutao Zhu Zhipeng Chen Wentong Chen Wayne Xin Zhao Xu Chen Yankai Lin Ji-Rong Wen Jiawei Han ELM 89 89 0 03 Nov 2023
Data Contamination Through the Lens of Time Manley Roberts Himanshu Thakur Christine Herlihy Colin White Samuel Dooley 60 30 0 16 Oct 2023
Questioning the Survey Responses of Large Language Models Ricardo Dominguez-Olmedo Moritz Hardt Celestine Mendler-Dünner 18 10 0 13 Jun 2023
GLM-130B: An Open Bilingual Pre-trained Model Aohan Zeng Xiao Liu Zhengxiao Du Zihan Wang Hanyu Lai ... Jidong Zhai Wenguang Chen Peng-Zhen Zhang Yuxiao Dong Jie Tang BDL LRM 237 840 0 05 Oct 2022
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 236 1,508 0 31 Dec 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 220 3,054 0 23 Jan 2020
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 294 6,003 0 20 Apr 2018