Humans or LLMs as the Judge? A Study on Judgement Biases

Humans or LLMs as the Judge? A Study on Judgement Biases

16 February 2024

Guiming Hardy Chen

Papers citing "Humans or LLMs as the Judge? A Study on Judgement Biases"

18 / 18 papers shown

Title
Real-World Gaps in AI Governance Research Ilan Strauss Isobel Moure Tim O'Reilly Sruly Rosenblat 52 0 0 30 Apr 2025
Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks Yixin Cao Shibo Hong X. Li Jiahao Ying Yubo Ma ... Juanzi Li Aixin Sun Xuanjing Huang Tat-Seng Chua Yu Jiang ALM ELM 84 0 0 26 Apr 2025
Regional Tiny Stories: Using Small Models to Compare Language Learning and Tokenizer Performance Nirvan Patil Malhar Abhay Inamdar Agnivo Gosai Guruprasad Pathak Anish Joshi Aryan Sagavekar Anish Joshirao Raj Abhijit Dandekar Rajat Dandekar Sreedath Panat 16 0 0 07 Apr 2025
Rate, Explain and Cite (REC): Enhanced Explanation and Attribution in Automatic Evaluation by Large Language Models Aliyah R. Hsu James Zhu Zhichao Wang Bin Bi Shubham Mehrotra ... Sougata Chaudhuri Regunathan Radhakrishnan S. Asur Claire Na Cheng Bin Yu ALM LRM 62 0 0 20 Feb 2025
Investigating Non-Transitivity in LLM-as-a-Judge Yi Xu Laura Ruis Tim Rocktaschel Robert Kirk 31 0 0 19 Feb 2025
Savaal: Scalable Concept-Driven Question Generation to Enhance Human Learning Kimia Noorbakhsh Joseph Chandler Pantea Karimi M. Alizadeh H. Balakrishnan LRM 31 1 0 18 Feb 2025
From Cool Demos to Production-Ready FMware: Core Challenges and a Technology Roadmap Gopi Krishnan Rajbahadur G. Oliva Dayi Lin Ahmed E. Hassan 33 0 0 28 Jan 2025
Surveying Attitudinal Alignment Between Large Language Models Vs. Humans Towards 17 Sustainable Development Goals Qingyang Wu Ying Xu Tingsong Xiao Yunze Xiao Yitong Li ... Yichi Zhang Shanghai Zhong Yuwei Zhang Wei Lu Yifan Yang 48 1 0 17 Jan 2025
WarriorCoder: Learning from Expert Battles to Augment Code Large Language Models Huawen Feng Pu Zhao Qingfeng Sun Can Xu Fangkai Yang ... Qianli Ma Qingwei Lin Saravan Rajmohan Dongmei Zhang Qi Zhang AAML ALM 57 0 0 23 Dec 2024
MIRAGE-Bench: Automatic Multilingual Benchmark Arena for Retrieval-Augmented Generation Systems Nandan Thakur Suleman Kazi Ge Luo Jimmy J. Lin Amin Ahmad VLM RALM 21 6 0 17 Oct 2024
One Language, Many Gaps: Evaluating Dialect Fairness and Robustness of Large Language Models in Reasoning Tasks Fangru Lin Shaoguang Mao Emanuele La Malfa Valentin Hofmann Adrian de Wynter Jing Yao Si-Qing Chen Michael Wooldridge Furu Wei Furu Wei 36 2 0 14 Oct 2024
RevisEval: Improving LLM-as-a-Judge via Response-Adapted References Qiyuan Zhang Yufei Wang Tiezheng YU Yuxin Jiang Chuhan Wu ... Xin Jiang Lifeng Shang Ruiming Tang Fuyuan Lyu Chen Ma 19 4 0 07 Oct 2024
Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown Xingzhou Lou Dong Yan Wei Shen Yuzi Yan Jian Xie Junge Zhang 41 21 0 01 Oct 2024
Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation Jasper Dekoninck Maximilian Baader Martin Vechev ALM 64 0 0 01 Sep 2024
Inverse Constitutional AI: Compressing Preferences into Principles Arduin Findeis Timo Kaufmann Eyke Hüllermeier Samuel Albanie Robert Mullins SyDa 41 8 0 02 Jun 2024
Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLP Anya Belz Craig Thomson Ehud Reiter Gavin Abercrombie J. Alonso-Moral ... Antonio Toral Xiao-Yi Wan Leo Wanner Lewis J. Watson Diyi Yang 58 23 0 02 May 2023
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 301 11,730 0 04 Mar 2022
Fine-Tuning Language Models from Human Preferences Daniel M. Ziegler Nisan Stiennon Jeff Wu Tom B. Brown Alec Radford Dario Amodei Paul Christiano G. Irving ALM 273 1,561 0 18 Sep 2019