Evaluating Large Language Models at Evaluating Instruction Following

11 October 2023

Papers citing "Evaluating Large Language Models at Evaluating Instruction Following"

33 / 33 papers shown

Title
Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators Yilun Zhou Austin Xu Peifeng Wang Caiming Xiong Shafiq R. Joty ELM ALM LRM 45 2 0 21 Apr 2025
Inference-Time Scaling for Generalist Reward Modeling Zijun Liu P. Wang R. Xu Shirong Ma Chong Ruan Peng Li Yang Janet Liu Y. Wu OffRL LRM 46 9 0 03 Apr 2025
Learning to Align Multi-Faceted Evaluation: A Unified and Robust Framework Kaishuai Xu Tiezheng YU Wenjun Hou Yi Cheng Liangyou Li Xin Jiang Lifeng Shang Q. Liu Wenjie Li ELM 66 0 0 26 Feb 2025
Order Matters: Investigate the Position Bias in Multi-constraint Instruction Following Jie Zeng Qianyu He Qingyu Ren Jiaqing Liang Yanghua Xiao Weikang Zhou Zeye Sun Fei Yu 84 0 0 24 Feb 2025
Varco Arena: A Tournament Approach to Reference-Free Benchmarking Large Language Models Seonil Son Ju-Min Oh Heegon Jin Cheolhun Jang Jeongbeom Jeong Kuntae Kim 39 0 0 20 Feb 2025
Aligning Black-box Language Models with Human Judgments Gerrit J. J. van den Burg Gen Suzuki Wei Liu Murat Sensoy ALM 71 0 0 07 Feb 2025
Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators Yinhong Liu Han Zhou Zhijiang Guo Ehsan Shareghi Ivan Vulić Anna Korhonen Nigel Collier ALM 128 67 0 20 Jan 2025
PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models Mingyang Song Zhaochen Su Xiaoye Qu Jiawei Zhou Yu-Xi Cheng LRM 45 29 0 06 Jan 2025
Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Models Yulei Qin Yuncheng Yang Pengcheng Guo Gang Li Hang Shao Yuchen Shi Zihan Xu Yun Gu Ke Li Xing Sun ALM 88 11 0 31 Dec 2024
SPaR: Self-Play with Tree-Search Refinement to Improve Instruction-Following in Large Language Models Jiale Cheng Xiao-Chang Liu C. Wang Xiaotao Gu Y. Lu Dan Zhang Yuxiao Dong J. Tang Hongning Wang Minlie Huang LRM 123 3 0 16 Dec 2024
Self-Generated Critiques Boost Reward Modeling for Language Models Yue Yu Zhengxing Chen Aston Zhang L Tan Chenguang Zhu ... Suchin Gururangan Chao-Yue Zhang Melanie Kambadur Dhruv Mahajan Rui Hou LRM ALM 87 14 0 25 Nov 2024
Interpreting Language Reward Models via Contrastive Explanations Junqi Jiang Tom Bewley Saumitra Mishra Freddy Lecue Manuela Veloso 74 0 0 25 Nov 2024
Smaller Large Language Models Can Do Moral Self-Correction Guangliang Liu Zhiyu Xue Rongrong Wang K. Johnson Kristen Marie Johnson LRM 23 0 0 30 Oct 2024
Are LLM-Judges Robust to Expressions of Uncertainty? Investigating the effect of Epistemic Markers on LLM-based Evaluation Dongryeol Lee Yerin Hwang Yongil Kim Joonsuk Park Kyomin Jung ELM 68 5 0 28 Oct 2024
Do LLMs estimate uncertainty well in instruction-following? Juyeon Heo Miao Xiong Christina Heinze-Deml Jaya Narain ELM 48 3 0 18 Oct 2024
MCQG-SRefine: Multiple Choice Question Generation and Evaluation with Iterative Self-Critique, Correction, and Comparison Feedback Zonghai Yao Aditya Parashar Huixue Zhou Won Seok Jang Feiyun Ouyang Zhichao Yang Hong-ye Yu ELM 37 2 0 17 Oct 2024
JudgeBench: A Benchmark for Evaluating LLM-based Judges Sijun Tan Siyuan Zhuang Kyle Montgomery William Y. Tang Alejandro Cuadron Chenguang Wang Raluca A. Popa Ion Stoica ELM ALM 51 36 0 16 Oct 2024
Evaluating the Instruction-following Abilities of Language Models using Knowledge Tasks Rudra Murthy Prince Kumar Praveen Venkateswaran Danish Contractor KELM ALM ELM 26 1 0 16 Oct 2024
RMB: Comprehensively Benchmarking Reward Models in LLM Alignment Enyu Zhou Guodong Zheng B. Wang Zhiheng Xi Shihan Dou ... Yurong Mou Rui Zheng Tao Gui Qi Zhang Xuanjing Huang ALM 54 14 0 13 Oct 2024
RevisEval: Improving LLM-as-a-Judge via Response-Adapted References Qiyuan Zhang Yufei Wang Tiezheng YU Yuxin Jiang Chuhan Wu ... Xin Jiang Lifeng Shang Ruiming Tang Fuyuan Lyu Chen Ma 26 4 0 07 Oct 2024
HelpSteer2-Preference: Complementing Ratings with Preferences Zhilin Wang Alexander Bukharin Olivier Delalleau Daniel Egert Gerald Shen Jiaqi Zeng Oleksii Kuchaiev Yi Dong ALM 42 39 0 02 Oct 2024
OffsetBias: Leveraging Debiased Data for Tuning Evaluators Junsoo Park Seungyeon Jwa Meiying Ren Daeyoung Kim Sanghyuk Choi ALM 29 29 0 09 Jul 2024
Evaluation of Instruction-Following Ability for Large Language Models on Story-Ending Generation Rem Hida Junki Ohmura Toshiyuki Sekiya ELM 27 0 0 24 Jun 2024
Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges Aman Singh Thakur Kartik Choudhary Venkat Srinik Ramayapally Sankaran Vaidyanathan Dieuwke Hupkes ELM ALM 45 55 0 18 Jun 2024
Testing and Understanding Erroneous Planning in LLM Agents through Synthesized User Inputs Zhenlan Ji Daoyuan Wu Pingchuan Ma Zongjie Li Shuai Wang LLMAG 40 3 0 27 Apr 2024
RewardBench: Evaluating Reward Models for Language Modeling Nathan Lambert Valentina Pyatkin Jacob Morrison Lester James Validad Miranda Bill Yuchen Lin ... Sachin Kumar Tom Zick Yejin Choi Noah A. Smith Hanna Hajishirzi ALM 74 210 0 20 Mar 2024
Can Large Language Models Be an Alternative to Human Evaluations? Cheng-Han Chiang Hung-yi Lee ALM LM&MA 209 559 0 03 May 2023
Instruction Tuning with GPT-4 Baolin Peng Chunyuan Li Pengcheng He Michel Galley Jianfeng Gao SyDa ALM LM&MA 157 576 0 06 Apr 2023
Can Large Language Models Truly Understand Prompts? A Case Study with Negated Prompts Joel Jang Seonghyeon Ye Minjoon Seo ELM LRM 87 64 0 26 Sep 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 303 11,730 0 04 Mar 2022
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason W. Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter F. Xia Ed H. Chi Quoc Le Denny Zhou LM&Ro LRM AI4CE ReLM 315 8,261 0 28 Jan 2022
$Understanding Dataset Difficulty with $\mathcal{V}$-Usable Information$ Understanding Dataset Difficulty with $\mathcal{V}$ -Usable Information Kawin Ethayarajh Yejin Choi Swabha Swayamdipta 154 157 0 16 Oct 2021
The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation Marzena Karpinska Nader Akoury Mohit Iyyer 204 106 0 14 Sep 2021