Large Language Models are not Fair Evaluators

29 May 2023

Peiyi Wang

Lei Li

Zefan Cai

Qi Liu

Zhifang Sui

Papers citing "Large Language Models are not Fair Evaluators"

50 / 71 papers shown

Title
ICon: In-Context Contribution for Automatic Data Selection Yixin Yang Qingxiu Dong Linli Yao Fangwei Zhu Zhifang Sui 41 0 0 08 May 2025
Improving Model Alignment Through Collective Intelligence of Open-Source LLMS Junlin Wang Roy Xie Shang Zhu Jue Wang Ben Athiwaratkun Bhuwan Dhingra S. Song Ce Zhang James Y. Zou ALM 24 0 0 05 May 2025
Automatic Legal Writing Evaluation of LLMs Ramon Pires Roseval Malaquias Junior Rodrigo Nogueira AILaw ELM 81 0 0 29 Apr 2025
Systematic Bias in Large Language Models: Discrepant Response Patterns in Binary vs. Continuous Judgment Tasks Yi-Long Lu C. Zhang Wei Wang 29 0 0 28 Apr 2025
Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks Yixin Cao Shibo Hong X. Li Jiahao Ying Yubo Ma ... Juanzi Li Aixin Sun Xuanjing Huang Tat-Seng Chua Yu Jiang ALM ELM 84 0 0 26 Apr 2025
Adversarial Attacks on LLM-as-a-Judge Systems: Insights from Prompt Injections Narek Maloyan Dmitry Namiot SILM AAML ELM 75 0 0 25 Apr 2025
AI Ethics and Social Norms: Exploring ChatGPT's Capabilities From What to How Omid Veisi Sasan Bahrami Roman Englert Claudia Müller 60 0 0 25 Apr 2025
LLM-as-a-Judge: Reassessing the Performance of LLMs in Extractive QA Xanh Ho Jiahao Huang Florian Boudin Akiko Aizawa ELM 29 0 0 16 Apr 2025
Towards Hierarchical Multi-Step Reward Models for Enhanced Reasoning in Large Language Models Teng Wang Zhangyi Jiang Zhenqi He Wenhan Yang Yanan Zheng Zeyu Li Zifan He Shenyang Tong Hailei Gong LRM 90 1 0 16 Mar 2025
Who Relies More on World Knowledge and Bias for Syntactic Ambiguity Resolution: Humans or LLMs? So Young Lee Russell Scheinberg Amber Shore Ameeta Agrawal 43 1 0 13 Mar 2025
Towards Efficient Educational Chatbots: Benchmarking RAG Frameworks Umar Ali Khan Ekram Khan Fiza Khan A. A. Moinuddin 48 0 0 02 Mar 2025
Learning to Align Multi-Faceted Evaluation: A Unified and Robust Framework Kaishuai Xu Tiezheng YU Wenjun Hou Yi Cheng Liangyou Li Xin Jiang Lifeng Shang Q. Liu Wenjie Li ELM 66 0 0 26 Feb 2025
Rate, Explain and Cite (REC): Enhanced Explanation and Attribution in Automatic Evaluation by Large Language Models Aliyah R. Hsu James Zhu Zhichao Wang Bin Bi Shubham Mehrotra ... Sougata Chaudhuri Regunathan Radhakrishnan S. Asur Claire Na Cheng Bin Yu ALM LRM 67 0 0 20 Feb 2025
Savaal: Scalable Concept-Driven Question Generation to Enhance Human Learning Kimia Noorbakhsh Joseph Chandler Pantea Karimi M. Alizadeh H. Balakrishnan LRM 44 1 0 18 Feb 2025
BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models Xu Huang Wenhao Zhu Hanxu Hu Conghui He Lei Li Shujian Huang Fei Yuan ELM 49 3 0 11 Feb 2025
Explaining Decisions of Agents in Mixed-Motive Games Maayan Orner Oleg Maksimov Akiva Kleinerman Charles Ortiz Sarit Kraus 86 0 0 28 Jan 2025
Panoramic Interests: Stylistic-Content Aware Personalized Headline Generation Junhong Lian Xiang Ao Xinyu Liu Yang Liu Qing He 32 0 0 21 Jan 2025
Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators Yinhong Liu Han Zhou Zhijiang Guo Ehsan Shareghi Ivan Vulić Anna Korhonen Nigel Collier ALM 128 64 0 20 Jan 2025
A 2-step Framework for Automated Literary Translation Evaluation: Its Promises and Pitfalls Sheikh Shafayat Dongkeun Yoon Woori Jang Jiwoo Choi Alice H. Oh Seohyon Jung 91 1 0 03 Jan 2025
PRD: Peer Rank and Discussion Improve Large Language Model based Evaluations Ruosen Li Teerth Patel Xinya Du LLMAG ALM 46 94 0 03 Jan 2025
Are You Doubtful? Oh, It Might Be Difficult Then! Exploring the Use of Model Uncertainty for Question Difficulty Estimation Leonidas Zotos H. Rijn Malvina Nissim 65 0 0 16 Dec 2024
MAG-V: A Multi-Agent Framework for Synthetic Data Generation and Verification Saptarshi Sengupta Kristal Curtis Akshay Mallipeddi Abhinav Mathur Joseph Ross Liang Gou Liang Gou LLMAG SyDa 100 1 0 28 Nov 2024
From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge Dawei Li Bohan Jiang Liangjie Huang Alimohammad Beigi Chengshuai Zhao ... Canyu Chen Tianhao Wu Kai Shu Lu Cheng Huan Liu ELM AILaw 106 61 0 25 Nov 2024
Are LLM-Judges Robust to Expressions of Uncertainty? Investigating the effect of Epistemic Markers on LLM-based Evaluation Dongryeol Lee Yerin Hwang Yongil Kim Joonsuk Park Kyomin Jung ELM 68 4 0 28 Oct 2024
PAPILLON: Privacy Preservation from Internet-based and Local Language Model Ensembles Li Siyan Vethavikashini Chithrra Raghuram Omar Khattab Julia Hirschberg Zhou Yu 21 7 0 22 Oct 2024
Do LLMs estimate uncertainty well in instruction-following? Juyeon Heo Miao Xiong Christina Heinze-Deml Jaya Narain ELM 46 2 0 18 Oct 2024
MIRAGE-Bench: Automatic Multilingual Benchmark Arena for Retrieval-Augmented Generation Systems Nandan Thakur Suleman Kazi Ge Luo Jimmy J. Lin Amin Ahmad VLM RALM 23 6 0 17 Oct 2024
MCQG-SRefine: Multiple Choice Question Generation and Evaluation with Iterative Self-Critique, Correction, and Comparison Feedback Zonghai Yao Aditya Parashar Huixue Zhou Won Seok Jang Feiyun Ouyang Zhichao Yang Hong-ye Yu ELM 37 2 0 17 Oct 2024
Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the data Florian E. Dorner Vivian Y. Nastl Moritz Hardt ELM ALM 33 5 0 17 Oct 2024
JudgeBench: A Benchmark for Evaluating LLM-based Judges Sijun Tan Siyuan Zhuang Kyle Montgomery William Y. Tang Alejandro Cuadron Chenguang Wang Raluca A. Popa Ion Stoica ELM ALM 47 35 0 16 Oct 2024
RevisEval: Improving LLM-as-a-Judge via Response-Adapted References Qiyuan Zhang Yufei Wang Tiezheng YU Yuxin Jiang Chuhan Wu ... Xin Jiang Lifeng Shang Ruiming Tang Fuyuan Lyu Chen Ma 26 4 0 07 Oct 2024
Position: LLM Unlearning Benchmarks are Weak Measures of Progress Pratiksha Thaker Shengyuan Hu Neil Kale Yash Maurya Zhiwei Steven Wu Virginia Smith MU 42 10 0 03 Oct 2024
Mitigating Copy Bias in In-Context Learning through Neuron Pruning Ameen Ali Lior Wolf Ivan Titov 27 2 0 02 Oct 2024
Mitigating Selection Bias with Node Pruning and Auxiliary Options Hyeong Kyu Choi Weijie Xu Chi Xue Stephanie Eckman Chandan K. Reddy 23 1 0 27 Sep 2024
Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation Jasper Dekoninck Maximilian Baader Martin Vechev ALM 87 0 0 01 Sep 2024
Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templates Hui Wei Shenghua He Tian Xia Andy H. Wong Jingyang Lin Mei Han Mei Han ALM ELM 49 22 0 23 Aug 2024
Automated Review Generation Method Based on Large Language Models Shican Wu Xiao Ma Dehui Luo Lulu Li Xiangcheng Shi ... Ran Luo Chunlei Pei Zhijian Zhao Zhi-Jian Zhao Jinlong Gong 69 0 0 30 Jul 2024
Controllable Contextualized Image Captioning: Directing the Visual Narrative through User-Defined Highlights Shunqi Mao Chaoyi Zhang Hang Su Hwanjun Song Igor Shalyminov Weidong Cai 26 1 0 16 Jul 2024
OffsetBias: Leveraging Debiased Data for Tuning Evaluators Junsoo Park Seungyeon Jwa Meiying Ren Daeyoung Kim Sanghyuk Choi ALM 29 29 0 09 Jul 2024
Can Model Uncertainty Function as a Proxy for Multiple-Choice Question Item Difficulty? Leonidas Zotos H. Rijn Malvina Nissim ELM 29 2 0 07 Jul 2024
MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and Efficient Evaluation Jinsheng Huang Liang Chen Taian Guo Fu Zeng Yusheng Zhao ... Wei Ju Luchen Liu Tianyu Liu Baobao Chang Ming Zhang 34 5 0 29 Jun 2024
CMMaTH: A Chinese Multi-modal Math Skill Evaluation Benchmark for Foundation Models Zhong-Zhi Li Ming-Liang Zhang Fei Yin Zhi-Long Ji Jin-Feng Bai Zhen-Ru Pan Fan-Hu Zeng Jian Xu Jia-Xin Zhang Cheng-Lin Liu ELM 23 10 0 28 Jun 2024
Large Language Models Assume People are More Rational than We Really are Ryan Liu Jiayi Geng Joshua C. Peterson Ilia Sucholutsky Thomas L. Griffiths 57 16 0 24 Jun 2024
DreamBench++: A Human-Aligned Benchmark for Personalized Image Generation Yuang Peng Yuxin Cui Haomiao Tang Zekun Qi Runpei Dong Jing Bai Chunrui Han Zheng Ge Xiangyu Zhang Shu-Tao Xia EGVM 64 30 0 24 Jun 2024
Crosslingual Capabilities and Knowledge Barriers in Multilingual Large Language Models Lynn Chua Badih Ghazi Yangsibo Huang Pritish Kamath Ravi Kumar Pasin Manurangsi Amer Sinha Chulin Xie Chiyuan Zhang 46 1 0 23 Jun 2024
Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges Aman Singh Thakur Kartik Choudhary Venkat Srinik Ramayapally Sankaran Vaidyanathan Dieuwke Hupkes ELM ALM 45 55 0 18 Jun 2024
Benchmark Data Contamination of Large Language Models: A Survey Cheng Xu Shuhao Guan Derek Greene Mohand-Tahar Kechadi ELM ALM 32 38 0 06 Jun 2024
Discovering Bias in Latent Space: An Unsupervised Debiasing Approach Dyah Adila Shuai Zhang Boran Han Yuyang Wang AAML LLMSV 19 6 0 05 Jun 2024
SLIDE: A Framework Integrating Small and Large Language Models for Open-Domain Dialogues Evaluation Kun Zhao Bohao Yang Chen Tang Chenghua Lin Liang Zhan 28 5 0 24 May 2024
SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modeling Xingzhou Lou Junge Zhang Jian Xie Lifeng Liu Dong Yan Kaiqi Huang 29 11 0 21 May 2024