Replacing Judges with Juries: Evaluating LLM Generations with a Panel of
Diverse Models

Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models

29 April 2024

Sebastian Hofstatter

Sophia Althammer

Aleksandra Piktus

Arkady Arkhangorodsky

Patrick Lewis

Papers citing "Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models"

13 / 13 papers shown

Title
Uncertainty Quantification for Language Models: A Suite of Black-Box, White-Box, LLM Judge, and Ensemble Scorers Dylan Bouchard Mohit Singh Chauhan HILM 70 0 0 27 Apr 2025
LLM-as-a-Judge: Reassessing the Performance of LLMs in Extractive QA Xanh Ho Jiahao Huang Florian Boudin Akiko Aizawa ELM 29 0 0 16 Apr 2025
The Relationship Between Reasoning and Performance in Large Language Models -- o3 (mini) Thinks Harder, Not Longer Marthe Ballon Andres Algaba Vincent Ginis LRM ReLM 36 4 0 24 Feb 2025
RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems Robert Friel Masha Belyi Atindriyo Sanyal 72 17 0 17 Jan 2025
AI Predicts AGI: Leveraging AGI Forecasting and Peer Review to Explore LLMs' Complex Reasoning Capabilities Fabrizio Davide Pietro Torre Andrea Gaggioli Andrea Gaggioli ELM 90 0 0 12 Dec 2024
JudgeBench: A Benchmark for Evaluating LLM-based Judges Sijun Tan Siyuan Zhuang Kyle Montgomery William Y. Tang Alejandro Cuadron Chenguang Wang Raluca A. Popa Ion Stoica ELM ALM 47 35 0 16 Oct 2024
TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generation Jonathan Cook Tim Rocktaschel Jakob Foerster Dennis Aumiller Alex Wang ALM 29 9 0 04 Oct 2024
PingPong: A Benchmark for Role-Playing Language Models with User Emulation and Multi-Model Evaluation Ilya Gusev LLMAG 52 3 0 10 Sep 2024
DCA-Bench: A Benchmark for Dataset Curation Agents Benhao Huang Yingzhuo Yu Jin Huang Xingjian Zhang Jiaqi Ma 23 1 0 11 Jun 2024
The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models Seungone Kim Juyoung Suk Ji Yong Cho Shayne Longpre Chaeeun Kim ... Sean Welleck Graham Neubig Moontae Lee Kyungjae Lee Minjoon Seo ELM ALM LM&MA 90 28 0 09 Jun 2024
(Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Texts Minghao Wu Jiahao Xu Yulin Yuan Gholamreza Haffari Longyue Wang Weihua Luo Kaifu Zhang LLMAG 114 22 0 20 May 2024
Latent Concept-based Explanation of NLP Models Xuemin Yu Fahim Dalvi Nadir Durrani Marzia Nouri Hassan Sajjad LRM FAtt 19 1 0 18 Apr 2024
JudgeLM: Fine-tuned Large Language Models are Scalable Judges Lianghui Zhu Xinggang Wang Xinlong Wang ELM ALM 54 103 0 26 Oct 2023