Generative Judge for Evaluating Alignment

9 October 2023

Papers citing "Generative Judge for Evaluating Alignment"

50 / 69 papers shown

Title
am-ELO: A Stable Framework for Arena-based LLM Evaluation Zirui Liu Jiatong Li Yan Zhuang Q. Liu Shuanghong Shen Jie Ouyang Mingyue Cheng Shijin Wang 30 0 0 06 May 2025
Uncertainty Quantification for Language Models: A Suite of Black-Box, White-Box, LLM Judge, and Ensemble Scorers Dylan Bouchard Mohit Singh Chauhan HILM 70 0 0 27 Apr 2025
Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks Yixin Cao Shibo Hong X. Li Jiahao Ying Yubo Ma ... Juanzi Li Aixin Sun Xuanjing Huang Tat-Seng Chua Yu Jiang ALM ELM 84 0 0 26 Apr 2025
Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators Yilun Zhou Austin Xu Peifeng Wang Caiming Xiong Shafiq R. Joty ELM ALM LRM 45 2 0 21 Apr 2025
PROMPTEVALS: A Dataset of Assertions and Guardrails for Custom Production Large Language Model Pipelines Reya Vir Shreya Shankar Harrison Chase Will Fu-Hinthorn Aditya G. Parameswaran AI4TS 32 0 0 20 Apr 2025
xVerify: Efficient Answer Verifier for Reasoning Model Evaluations Ding Chen Qingchen Yu P. Wang W. Zhang Bo Tang Feiyu Xiong X. Li Minchuan Yang Z. Li ALM LRM 28 2 0 14 Apr 2025
A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Future Jialun Zhong Wei Shen Yanzeng Li Songyang Gao Hua Lu Yicheng Chen Yang Zhang Wei Zhou Jinjie Gu Lei Zou LRM 38 2 0 12 Apr 2025
Large Language Models as Span Annotators Zdeněk Kasner Vilém Zouhar Patrícia Schmidtová Ivan Kartáč Kristýna Onderková Ondřej Plátek Dimitra Gkatzia Saad Mahamood Ondrej Dusek Simone Balloccu ALM 30 0 0 11 Apr 2025
AgentAda: Skill-Adaptive Data Analytics for Tailored Insight Discovery Amirhossein Abaskohi A. Ramesh Shailesh Nanisetty Chirag Goel David Vazquez Christopher Pal Spandana Gella Giuseppe Carenini I. Laradji 29 0 0 10 Apr 2025
Benchmarking Adversarial Robustness to Bias Elicitation in Large Language Models: Scalable Automated Assessment with LLM-as-a-Judge Riccardo Cantini A. Orsino Massimo Ruggiero Domenico Talia AAML ELM 40 0 0 10 Apr 2025
TALE: A Tool-Augmented Framework for Reference-Free Evaluation of Large Language Models Sher Badshah Ali Emami Hassan Sajjad LLMAG ELM 43 0 0 10 Apr 2025
Inference-Time Scaling for Generalist Reward Modeling Zijun Liu P. Wang R. Xu Shirong Ma Chong Ruan Peng Li Yang Janet Liu Y. Wu OffRL LRM 46 9 0 03 Apr 2025
Multi-Agent LLM Judge: automatic personalized LLM judge design for evaluating natural language generation applications Hongliu Cao Ilias Driouich Robin Singh Eoin Thomas ELM 36 0 0 01 Apr 2025
Dancing with Critiques: Enhancing LLM Reasoning with Stepwise Natural Language Self-Critique Y. Li Jiahao Xu Tian Liang Xingyu Chen Zhiwei He ... Rui Wang Z. Zhang Zhaopeng Tu Haitao Mi Dong Yu LRM 43 1 0 21 Mar 2025
Is Your Video Language Model a Reliable Judge? M. Liu Wensheng Zhang 59 1 0 07 Mar 2025
SePer: Measure Retrieval Utility Through The Lens Of Semantic Perplexity Reduction Lu Dai Yijie Xu Jinhui Ye Hao Liu Hui Xiong 3DV RALM 76 2 0 03 Mar 2025
Self-Rationalization in the Wild: A Large Scale Out-of-Distribution Evaluation on NLI-related tasks Jing Yang Max Glockner Anderson de Rezende Rocha Iryna Gurevych LRM 62 1 0 07 Feb 2025
Evaluation of Large Language Models via Coupled Token Generation N. C. Benz Stratis Tsirtsis Eleni Straitouri Ivi Chatzi Ander Artola Velasco Suhas Thejaswi Manuel Gomez Rodriguez 46 0 0 03 Feb 2025
Improving Video Generation with Human Feedback Jie Liu Gongye Liu Jiajun Liang Ziyang Yuan Xiaokun Liu ... Pengfei Wan Di Zhang Kun Gai Yujiu Yang Wanli Ouyang VGen EGVM 56 13 0 23 Jan 2025
Refining Input Guardrails: Enhancing LLM-as-a-Judge Efficiency Through Chain-of-Thought Fine-Tuning and Alignment Melissa Kazemi Rad Huy Nghiem Andy Luo Sahil Wadhwa Mohammad Sorower Stephen Rawls AAML 91 2 0 22 Jan 2025
ACE- $M^3$ : Automatic Capability Evaluator for Multimodal Medical Models Xiechi Zhang Shunfan Zheng Linlin Wang Gerard de Melo Zhu Cao Xiaoling Wang Liang He ELM 99 0 0 16 Dec 2024
The Superalignment of Superhuman Intelligence with Large Language Models Minlie Huang Yingkang Wang Shiyao Cui Pei Ke J. Tang 103 1 0 15 Dec 2024
LLM Augmentations to support Analytical Reasoning over Multiple Documents Raquib Bin Yousuf Nicholas Defelice Mandar Sharma Shengzhe Xu Naren Ramakrishnan 59 2 0 25 Nov 2024
Self-Generated Critiques Boost Reward Modeling for Language Models Yue Yu Zhengxing Chen Aston Zhang L Tan Chenguang Zhu ... Suchin Gururangan Chao-Yue Zhang Melanie Kambadur Dhruv Mahajan Rui Hou LRM ALM 87 14 0 25 Nov 2024
From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Set M. Finkelstein Dan Deutsch Parker Riley Juraj Juraska Geza Kovacs Markus Freitag 74 0 0 23 Nov 2024
JailbreakLens: Interpreting Jailbreak Mechanism in the Lens of Representation and Circuit Zeqing He Zhibo Wang Zhixuan Chu Huiyu Xu Rui Zheng Kui Ren Chun Chen 52 3 0 17 Nov 2024
CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and Evolution Maosong Cao Alexander Lam Haodong Duan Hongwei Liu S. Zhang Kai Chen AILaw ELM 37 11 0 21 Oct 2024
JudgeBench: A Benchmark for Evaluating LLM-based Judges Sijun Tan Siyuan Zhuang Kyle Montgomery William Y. Tang Alejandro Cuadron Chenguang Wang Raluca A. Popa Ion Stoica ELM ALM 51 36 0 16 Oct 2024
Varying Shades of Wrong: Aligning LLMs with Wrong Answers Only Jihan Yao Wenxuan Ding Shangbin Feng Lucy Lu Wang Yulia Tsvetkov 25 0 0 14 Oct 2024
4-LEGS: 4D Language Embedded Gaussian Splatting Gal Fiebelman Tamir Cohen Ayellet Morgenstern Peter Hedman Hadar Averbuch-Elor 3DGS 41 1 0 14 Oct 2024
RevisEval: Improving LLM-as-a-Judge via Response-Adapted References Qiyuan Zhang Yufei Wang Tiezheng YU Yuxin Jiang Chuhan Wu ... Xin Jiang Lifeng Shang Ruiming Tang Fuyuan Lyu Chen Ma 26 4 0 07 Oct 2024
Take It Easy: Label-Adaptive Self-Rationalization for Fact Verification and Explanation Generation Jing Yang Anderson de Rezende Rocha 23 1 0 05 Oct 2024
Beyond Scalar Reward Model: Learning Generative Judge from Preference Data Ziyi Ye Xiangsheng Li Qiuchi Li Qingyao Ai Yujia Zhou Wei Shen Dong Yan Yiqun Liu 48 10 0 01 Oct 2024
Direct Judgement Preference Optimization Peifeng Wang Austin Xu Yilun Zhou Caiming Xiong Shafiq Joty ELM 37 12 0 23 Sep 2024
Aligning Language Models Using Follow-up Likelihood as Reward Signal Chen Zhang Dading Chong Feng Jiang Chengguang Tang Anningzhe Gao Guohua Tang Haizhou Li ALM 29 2 0 20 Sep 2024
Towards a Unified View of Preference Learning for Large Language Models: A Survey Bofei Gao Feifan Song Yibo Miao Zefan Cai Z. Yang ... Houfeng Wang Zhifang Sui Peiyi Wang Baobao Chang Baobao Chang 41 11 0 04 Sep 2024
Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templates Hui Wei Shenghua He Tian Xia Andy H. Wong Jingyang Lin Mei Han Mei Han ALM ELM 59 23 0 23 Aug 2024
Critique-out-Loud Reward Models Zachary Ankner Mansheej Paul Brandon Cui Jonathan D. Chang Prithviraj Ammanabrolu ALM LRM 25 26 0 21 Aug 2024
Evaluating the Evaluator: Measuring LLMs' Adherence to Task Evaluation Instructions Bhuvanashree Murugadoss Christian Poelitz Ian Drosos Vu Le Nick McKenna Carina Negreanu Chris Parnin Advait Sarkar ELM ALM 35 12 0 16 Aug 2024
CLAVE: An Adaptive Framework for Evaluating Values of LLM Generated Responses Jing Yao Xiaoyuan Yi Xing Xie ELM ALM 36 7 0 15 Jul 2024
Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges Aman Singh Thakur Kartik Choudhary Venkat Srinik Ramayapally Sankaran Vaidyanathan Dieuwke Hupkes ELM ALM 45 55 0 18 Jun 2024
A Survey on Human Preference Learning for Large Language Models Ruili Jiang Kehai Chen Xuefeng Bai Zhixuan He Juntao Li Muyun Yang Tiejun Zhao Liqiang Nie Min Zhang 39 8 0 17 Jun 2024
Incentivizing Quality Text Generation via Statistical Contracts Eden Saig Ohad Einav Inbal Talgam-Cohen 31 2 0 17 Jun 2024
The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models Seungone Kim Juyoung Suk Ji Yong Cho Shayne Longpre Chaeeun Kim ... Sean Welleck Graham Neubig Moontae Lee Kyungjae Lee Minjoon Seo ELM ALM LM&MA 97 29 0 09 Jun 2024
Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas Chengyuan Deng Yiqun Duan Xin Jin Heng Chang Yijun Tian ... Kuofeng Gao Sihong He Jun Zhuang Lu Cheng Haohan Wang AILaw 38 16 0 08 Jun 2024
Dishonesty in Helpful and Harmless Alignment Youcheng Huang Jingkun Tang Duanyu Feng Zheng-Wei Zhang Wenqiang Lei Jiancheng Lv Anthony G. Cohn LLMSV 33 3 0 04 Jun 2024
Improving Reward Models with Synthetic Critiques Zihuiwen Ye Fraser Greenlee-Scott Max Bartolo Phil Blunsom Jon Ander Campos Matthias Gallé ALM SyDa LRM 38 17 0 31 May 2024
SED: Self-Evaluation Decoding Enhances Large Language Models for Better Generation Ziqin Luo Haixia Han Haokun Zhao Guochao Jiang Chengyu Du Tingyun Li Jiaqing Liang Deqing Yang Yanghua Xiao 46 3 0 26 May 2024
Self-Play Preference Optimization for Language Model Alignment Yue Wu Zhiqing Sun Huizhuo Yuan Kaixuan Ji Yiming Yang Quanquan Gu 23 113 0 01 May 2024
Sample-Efficient Human Evaluation of Large Language Models via Maximum Discrepancy Competition Kehua Feng Keyan Ding Kede Ma Zhihua Wang Qiang Zhang Huajun Chen 19 10 0 10 Apr 2024