From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline

17 June 2024

Joseph E. Gonzalez

Ion Stoica

ALM

ArXiv PDF HTML

Papers citing "From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline"

30 / 30 papers shown

Title
Improving Model Alignment Through Collective Intelligence of Open-Source LLMS Junlin Wang Roy Xie Shang Zhu Jue Wang Ben Athiwaratkun Bhuwan Dhingra S. Song Ce Zhang James Y. Zou ALM 24 0 0 05 May 2025
Exploring the Potential of Offline RL for Reasoning in LLMs: A Preliminary Study Xiaoyu Tian Sitong Zhao Haotian Wang Shuaiting Chen Yiping Peng Yunjie Ji Han Zhao Xiangang Li OffRL LRM 24 0 0 04 May 2025
Phi-4-reasoning Technical Report Marah Abdin Sahaj Agarwal Ahmed Hassan Awadallah Vidhisha Balachandran Harkirat Singh Behl ... Vaishnavi Shrivastava Vibhav Vineet Yue Wu Safoora Yousefi Guoqing Zheng ReLM LRM 74 0 0 30 Apr 2025
Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks Yixin Cao Shibo Hong X. Li Jiahao Ying Yubo Ma ... Juanzi Li Aixin Sun Xuanjing Huang Tat-Seng Chua Yu Jiang ALM ELM 84 0 0 26 Apr 2025
Pre-DPO: Improving Data Utilization in Direct Preference Optimization Using a Guiding Reference Model Junshu Pan Wei Shen Shulin Huang Qiji Zhou Yue Zhang 69 0 0 22 Apr 2025
FuseRL: Dense Preference Optimization for Heterogeneous Model Fusion Longguang Zhong Fanqi Wan Ziyi Yang Guosheng Liang Tianyuan Shi Xiaojun Quan MoMe 45 0 0 09 Apr 2025
A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applications Jian-Yu Guan J. Wu J. Li Chuanqi Cheng Wei Yu Wu LM&MA 69 0 0 21 Mar 2025
WildIFEval: Instruction Following in the Wild Gili Lior Asaf Yehudai Ariel Gera L. Ein-Dor 59 0 0 09 Mar 2025
Every FLOP Counts: Scaling a 300B Mixture-of-Experts LING LLM without Premium GPUs Ling Team B. Zeng C. Huang Chao Zhang Changxin Tian ... Zhaoxin Huan Zujie Wen Zhenhang Sun Zhuoxuan Du Z. He MoE ALM 87 2 0 07 Mar 2025
OmniAlign-V: Towards Enhanced Alignment of MLLMs with Human Preference Xiangyu Zhao Shengyuan Ding Zicheng Zhang Haian Huang Maosong Cao ... Wenhai Wang Guangtao Zhai Haodong Duan Hua Yang Kai Chen 78 6 0 25 Feb 2025
Synthesizing Post-Training Data for LLMs through Multi-Agent Simulation Shuo Tang Xianghe Pang Zexi Liu Bohan Tang Rui Ye Xiaowen Dong Y. Wang Yanfeng Wang S. Chen SyDa LLMAG 106 3 0 21 Feb 2025
BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models Xu Huang Wenhao Zhu Hanxu Hu Conghui He Lei Li Shujian Huang Fei Yuan ELM 35 3 0 11 Feb 2025
WarriorCoder: Learning from Expert Battles to Augment Code Large Language Models Huawen Feng Pu Zhao Qingfeng Sun Can Xu Fangkai Yang ... Qianli Ma Qingwei Lin Saravan Rajmohan Dongmei Zhang Qi Zhang AAML ALM 57 0 0 23 Dec 2024
Yi-Lightning Technical Report 01. AI : Alan Wake Albert Wang Bei Chen ... Yuxuan Sha Zhaodong Yan Zhiyuan Liu Zirui Zhang Zonghong Dai OSLM 82 3 0 02 Dec 2024
Puzzle: Distillation-Based NAS for Inference-Optimized LLMs Akhiad Bercovich Tomer Ronen Talor Abramovich Nir Ailon Nave Assaf ... Ido Shahaf Oren Tropp Omer Ullman Argov Ran Zilberstein Ran El-Yaniv 65 1 0 28 Nov 2024
From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge Dawei Li Bohan Jiang Liangjie Huang Alimohammad Beigi Chengshuai Zhao ... Canyu Chen Tianhao Wu Kai Shu Lu Cheng Huan Liu ELM AILaw 96 61 0 25 Nov 2024
UTMath: Math Evaluation with Unit Test via Reasoning-to-Coding Thoughts Bo Yang Qingping Yang Runtao Liu Runtao Liu LRM ReLM ELM AIMat 49 1 0 11 Nov 2024
$f$ -PO: Generalizing Preference Optimization with $f$ -divergence Minimization Jiaqi Han Mingjian Jiang Yuxuan Song J. Leskovec Stefano Ermon 32 3 0 29 Oct 2024
Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model Wenhong Zhu Zhiwei He Xiaofeng Wang Pengfei Liu Rui Wang OSLM 37 3 0 24 Oct 2024
Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the data Florian E. Dorner Vivian Y. Nastl Moritz Hardt ELM ALM 22 5 0 17 Oct 2024
JudgeBench: A Benchmark for Evaluating LLM-based Judges Sijun Tan Siyuan Zhuang Kyle Montgomery William Y. Tang Alejandro Cuadron Chenguang Wang Raluca A. Popa Ion Stoica ELM ALM 37 35 0 16 Oct 2024
One Language, Many Gaps: Evaluating Dialect Fairness and Robustness of Large Language Models in Reasoning Tasks Fangru Lin Shaoguang Mao Emanuele La Malfa Valentin Hofmann Adrian de Wynter Jing Yao Si-Qing Chen Michael Wooldridge Furu Wei Furu Wei 36 2 0 14 Oct 2024
RMB: Comprehensively Benchmarking Reward Models in LLM Alignment Enyu Zhou Guodong Zheng B. Wang Zhiheng Xi Shihan Dou ... Yurong Mou Rui Zheng Tao Gui Qi Zhang Xuanjing Huang ALM 49 13 0 13 Oct 2024
Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirements Jingyu Zhang Ahmed Elgohary Ahmed Magooda Daniel Khashabi Benjamin Van Durme 33 2 0 11 Oct 2024
Reward-Augmented Data Enhances Direct Preference Alignment of LLMs Shenao Zhang Zhihan Liu Boyi Liu Y. Zhang Yingxiang Yang Y. Liu Liyu Chen Tao Sun Z. Wang 82 2 0 10 Oct 2024
ConsistencyTrack: A Robust Multi-Object Tracker with a Generation Strategy of Consistency Model Lifan Jiang Zhihui Wang Siqi Yin Guangxiao Ma Peng Zhang Boxi Wu DiffM 43 2 0 28 Aug 2024
Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templates Hui Wei Shenghua He Tian Xia Andy H. Wong Jingyang Lin Mei Han Mei Han ALM ELM 42 22 0 23 Aug 2024
The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation Marzena Karpinska Nader Akoury Mohit Iyyer 198 106 0 14 Sep 2021
Extracting Training Data from Large Language Models Nicholas Carlini Florian Tramèr Eric Wallace Matthew Jagielski Ariel Herbert-Voss ... Tom B. Brown D. Song Ulfar Erlingsson Alina Oprea Colin Raffel MLAU SILM 261 1,386 0 14 Dec 2020
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 294 6,003 0 20 Apr 2018