JudgeBench: A Benchmark for Evaluating LLM-based Judges

16 October 2024

Ion Stoica

Papers citing "JudgeBench: A Benchmark for Evaluating LLM-based Judges"

22 / 22 papers shown

Title
Thoughts without Thinking: Reconsidering the Explanatory Value of Chain-of-Thought Reasoning in LLMs through Agentic Pipelines R. Manuvinakurike Emanuel Moss E. A. Watkins Saurav Sahay G. Raffa L. Nachman LRM 17 0 0 01 May 2025
Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems Shaokun Zhang Ming Yin Jieyu Zhang J. H. Liu Zhiguang Han ... Beibin Li Chi Wang H. Wang Y. Chen Qingyun Wu 47 0 0 30 Apr 2025
Leveraging LLMs as Meta-Judges: A Multi-Agent Framework for Evaluating LLM Judgments Y. Li Jama Hussein Mohamud Chongren Sun Di Wu Benoit Boulet LLMAG ELM 60 0 0 23 Apr 2025
Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators Yilun Zhou Austin Xu Peifeng Wang Caiming Xiong Shafiq R. Joty ELM ALM LRM 41 1 0 21 Apr 2025
Efficient MAP Estimation of LLM Judgment Performance with Prior Transfer Huaizhi Qu Inyoung Choi Zhen Tan Song Wang Sukwon Yun Qi Long Faizan Siddiqui Kwonjoon Lee Tianlong Chen 39 0 0 17 Apr 2025
A Multi-Model Adaptation of Speculative Decoding for Classification Somnath Roy Padharthi Sreekar Srivatsa Narasimha Anubhav Anand 34 0 0 23 Mar 2025
Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settings Austin Xu Srijan Bansal Yifei Ming Semih Yavuz Shafiq R. Joty ELM 87 2 0 19 Mar 2025
UPME: An Unsupervised Peer Review Framework for Multimodal Large Language Model Evaluation Qihui Zhang Munan Ning Zheyuan Liu Yanbo Wang Jiayi Ye Yue Huang Shuo Yang Xiao Chen Y. Song Li Yuan LRM 53 0 0 19 Mar 2025
ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learning Ziyu Wan Yunxiang Li Y. Song Hanjing Wang Linyi Yang Mark W. Schmidt J. Wang Weinan Zhang Shuyue Hu Ying Wen LLMAG KELM LRM AI4CE 81 5 0 12 Mar 2025
GRP: Goal-Reversed Prompting for Zero-Shot Evaluation with LLMs Mingyang Song Mao Zheng Xuan Luo LRM 53 0 0 08 Mar 2025
No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding Michael Krumdick Charles Lovering Varshini Reddy Seth Ebner Chris Tanner ALM ELM 43 2 0 07 Mar 2025
Improving LLM-as-a-Judge Inference with the Judgment Distribution Victor Wang Michael J.Q. Zhang Eunsol Choi 46 0 0 04 Mar 2025
LangProBe: a Language Programs Benchmark Shangyin Tan Lakshya A Agrawal Arnav Singhvi Liheng Lai Michael J Ryan Dan Klein Omar Khattab Koushik Sen Matei A. Zaharia 59 0 0 27 Feb 2025
Can Large Language Models Detect Errors in Long Chain-of-Thought Reasoning? Yancheng He Shilong Li J. Liu Weixun Wang Xingyuan Bu ... Zhongyuan Peng Z. Zhang Zhicheng Zheng Wenbo Su Bo Zheng ELM LRM 60 6 0 26 Feb 2025
Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systems Hao Peng Y. Qi Xiaozhi Wang Zijun Yao Bin Xu Lei Hou Juanzi Li ALM LRM 52 4 0 26 Feb 2025
CodeCriticBench: A Holistic Code Critique Benchmark for Large Language Models Alexander Zhang Marcus Dong J. H. Liu W. Zhang Yejie Wang ... Yancheng He K. Deng Wangchunshu Zhou Wenhao Huang Z. Zhang LRM 44 2 0 23 Feb 2025
Critical-Questions-of-Thought: Steering LLM reasoning with Argumentative Querying Federico Castagna I. Sassoon Simon Parsons LRM 80 0 0 19 Dec 2024
JuStRank: Benchmarking LLM Judges for System Ranking Ariel Gera Odellia Boni Yotam Perlitz Roy Bar-Haim Lilach Eden Asaf Yehudai ALM ELM 90 2 0 12 Dec 2024
VLRewardBench: A Challenging Benchmark for Vision-Language Generative Reward Models Lei Li Y. X. Wei Zhihui Xie Xuqing Yang Yifan Song ... Tianyu Liu Sujian Li Bill Yuchen Lin Lingpeng Kong Q. Liu CoGe VLM 107 24 0 26 Nov 2024
From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge Dawei Li Bohan Jiang Liangjie Huang Alimohammad Beigi Chengshuai Zhao ... Canyu Chen Tianhao Wu Kai Shu Lu Cheng Huan Liu ELM AILaw 99 61 0 25 Nov 2024
RevisEval: Improving LLM-as-a-Judge via Response-Adapted References Qiyuan Zhang Yufei Wang Tiezheng YU Yuxin Jiang Chuhan Wu ... Xin Jiang Lifeng Shang Ruiming Tang Fuyuan Lyu Chen Ma 26 4 0 07 Oct 2024
DHP Benchmark: Are LLMs Good NLG Evaluators? Yicheng Wang Jiayi Yuan Yu-Neng Chuang Zhuoer Wang Yingchi Liu Mark Cusick Param Kulkarni Zhengping Ji Yasser Ibrahim Xia Hu LM&MA ELM 31 3 0 25 Aug 2024