WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild

7 June 2024

Bill Yuchen Lin

Yuntian Deng

Khyathi Raghavi Chandu

Faeze Brahman

Abhilasha Ravichander

Yejin Choi

Papers citing "WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild"

50 / 55 papers shown

Title
R-Bench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation Meng-Hao Guo Jiajun Xu Yi Zhang Jiaxi Song Haoyang Peng ... Yongming Rao Houwen Peng Han Hu Gordon Wetzstein Shi-Min Hu ELM LRM 52 0 0 04 May 2025
Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks Yixin Cao Shibo Hong X. Li Jiahao Ying Yubo Ma ... Juanzi Li Aixin Sun Xuanjing Huang Tat-Seng Chua Yu Jiang ALM ELM 84 0 0 26 Apr 2025
What's the Difference? Supporting Users in Identifying the Effects of Prompt and Model Changes Through Token Patterns Michael A. Hedderich Anyi Wang Raoyuan Zhao Florian Eichin Barbara Plank 30 0 0 22 Apr 2025
EvalAgent: Discovering Implicit Evaluation Criteria from the Web Manya Wadhwa Zayne Sprague Chaitanya Malaviya Philippe Laban Junyi Jessy Li Greg Durrett 27 0 0 21 Apr 2025
MIG: Automatic Data Selection for Instruction Tuning by Maximizing Information Gain in Semantic Space Yicheng Chen Yining Li Kai Hu Zerun Ma Haochen Ye Kai Chen 27 0 0 18 Apr 2025
DICE: A Framework for Dimensional and Contextual Evaluation of Language Models Aryan Shrivastava Paula Akemi Aoyagui 26 0 0 14 Apr 2025
Genius: A Generalizable and Purely Unsupervised Self-Training Framework For Advanced Reasoning FangZhi Xu Hang Yan Chang Ma Haiteng Zhao Qiushi Sun Kanzhi Cheng Junxian He Jun Liu Zhiyong Wu LRM 24 1 0 11 Apr 2025
AIR: A Systematic Analysis of Annotations, Instructions, and Response Pairs in Preference Dataset Bingxiang He Wenbin Zhang Jiaxi Song Cheng Qian Z. Fu ... Hui Xue Ganqu Cui Wanxiang Che Zhiyuan Liu Maosong Sun 29 0 0 04 Apr 2025
ChatBench: From Static Benchmarks to Human-AI Evaluation Serina Chang Ashton Anderson Jake M. Hofman ELM AI4MH 57 2 0 22 Mar 2025
The Emperor's New Clothes in Benchmarking? A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination Yifan Sun Han Wang Dongbai Li Gang Wang Huan Zhang AAML 53 0 0 20 Mar 2025
Synthetic Clarification and Correction Dialogues about Data-Centric Tasks -- A Teacher-Student Approach Christian Poelitz Nick McKenna 43 1 0 18 Mar 2025
CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era Kanzhi Cheng Wenpo Song Jiaxin Fan Zheng Ma Qiushi Sun Fangzhi Xu Chenyang Yan Nuo Chen Jianbing Zhang Jiajun Chen MLLM VLM 45 1 0 16 Mar 2025
SmartBench: Is Your LLM Truly a Good Chinese Smartphone Assistant? Xudong Lu Haohao Gao Renshou Wu Shuai Ren Xiaoxin Chen Hongsheng Li Fangyuan Li ELM 49 0 0 08 Mar 2025
RocketEval: Efficient Automated LLM Evaluation via Grading Checklist Tianjun Wei Wei Wen Ruizhi Qiao Xing Sun Jianghong Ma ALM ELM 45 1 0 07 Mar 2025
Toward an Evaluation Science for Generative AI Systems Laura Weidinger Deb Raji Hanna M. Wallach Margaret Mitchell Angelina Wang Olawale Salaudeen Rishi Bommasani Sayash Kapoor Deep Ganguli Sanmi Koyejo EGVM ELM 62 3 0 07 Mar 2025
Revitalizing Saturated Benchmarks: A Weighted Metric Approach for Differentiating Large Language Model Performance Bryan Etzine Masoud Hashemi Nishanth Madhusudhan Sagar Davasam Roshnee Sharma Sathwik Tejaswi Madhusudhan Vikas Yadav 39 0 0 07 Mar 2025
Improving LLM-as-a-Judge Inference with the Judgment Distribution Victor Wang Michael J.Q. Zhang Eunsol Choi 53 0 0 04 Mar 2025
EAIRA: Establishing a Methodology for Evaluating AI Models as Scientific Research Assistants Franck Cappello Sandeep Madireddy Robert Underwood N. Getty Nicholas Chia ... M. Rafique Eliu A. Huerta B. Li Ian Foster Rick L. Stevens 66 1 0 27 Feb 2025
Kanana: Compute-efficient Bilingual Language Models Kanana LLM Team Yunju Bak Hojin Lee Minho Ryu Jiyeon Ham ... Daniel Lee Minchul Lee M. Lee Shinbok Lee Gaeun Seo 80 1 0 26 Feb 2025
Mind the Gap! Static and Interactive Evaluations of Large Audio Models Minzhi Li William B. Held Michael Joseph Ryan Kunat Pipatanakul Potsawee Manakul Hao Zhu Diyi Yang AuLLM ALM 56 0 0 21 Feb 2025
Rate, Explain and Cite (REC): Enhanced Explanation and Attribution in Automatic Evaluation by Large Language Models Aliyah R. Hsu James Zhu Zhichao Wang Bin Bi Shubham Mehrotra ... Sougata Chaudhuri Regunathan Radhakrishnan S. Asur Claire Na Cheng Bin Yu ALM LRM 67 0 0 20 Feb 2025
Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMs Longxu Dou Qian Liu Fan Zhou Changyu Chen Zili Wang ... Tianyu Pang Chao Du Xinyi Wan Wei Lu Min Lin 82 1 0 18 Feb 2025
Beyond the Singular: The Essential Role of Multiple Generations in Effective Benchmark Evaluation and Analysis Wenbo Zhang Hengrui Cai Wenyu Chen 77 0 0 17 Feb 2025
SeaExam and SeaBench: Benchmarking LLMs with Local Multilingual Questions in Southeast Asia Chaoqun Liu Wenxuan Zhang Jiahao Ying Mahani Aljunied Anh Tuan Luu Lidong Bing ELM 44 1 0 10 Feb 2025
Can LLMs Rank the Harmfulness of Smaller LLMs? We are Not There Yet Berk Atil Vipul Gupta Sarkar Snigdha Sarathi Das R. Passonneau 81 0 0 07 Feb 2025
ELITR-Bench: A Meeting Assistant Benchmark for Long-Context Language Models Thibaut Thonet Jos Rozen Laurent Besacier RALM 129 2 0 20 Jan 2025
Yi-Lightning Technical Report 01. AI : Alan Wake Albert Wang Bei Chen ... Yuxuan Sha Zhaodong Yan Zhiyuan Liu Zirui Zhang Zonghong Dai OSLM 97 3 0 02 Dec 2024
A Benchmark for Long-Form Medical Question Answering Pedram Hosseini Jessica M. Sin Bing Ren Bryceton G. Thomas Elnaz Nouri Ali Farahanchi Saeed Hassanpour ELM LM&MA AI4MH 35 0 0 14 Nov 2024
Project MPG: towards a generalized performance benchmark for LLM capabilities Lucas Spangher Tianle Li William Arnold Nick Masiewicki Xerxes Dotiwalla Rama Parusmathi Peter Grabowski Eugene Ie Dan Gruhl 36 0 0 28 Oct 2024
Limitations of the LLM-as-a-Judge Approach for Evaluating LLM Outputs in Expert Knowledge Tasks Annalisa Szymanski Noah Ziems Heather A. Eicher-Miller T. Li Meng-Long Jiang Ronald A Metoyer ALM ELM 36 19 0 26 Oct 2024
Benchmarking Foundation Models on Exceptional Cases: Dataset Creation and Validation Suho Kang Jungyang Park Joonseo Ha SoMin Kim JinHyeong Kim Subeen Park Kyungwoo Song LRM 16 0 0 23 Oct 2024
MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Models Guijin Son Dongkeun Yoon Juyoung Suk Javier Aula-Blasco Mano Aslan Vu Trong Kim Shayekh Bin Islam Jaume Prats-Cristià Lucía Tormo-Bañuelos Seungone Kim ELM LRM 25 0 0 23 Oct 2024
CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and Evolution Maosong Cao Alexander Lam Haodong Duan Hongwei Liu S. Zhang Kai Chen AILaw ELM 37 11 0 21 Oct 2024
Diverging Preferences: When do Annotators Disagree and do Models Know? Michael J.Q. Zhang Zhilin Wang Jena D. Hwang Yi Dong Olivier Delalleau Yejin Choi Eunsol Choi Xiang Ren Valentina Pyatkin 19 7 0 18 Oct 2024
FB-Bench: A Fine-Grained Multi-Task Benchmark for Evaluating LLMs' Responsiveness to Human Feedback Y. Li Miao Zheng Fan Yang Guosheng Dong Bin Cui Weipeng Chen Zenan Zhou Wentao Zhang ALM 32 5 0 12 Oct 2024
Packing Analysis: Packing Is More Appropriate for Large Models or Datasets in Supervised Fine-tuning Shuhe Wang Guoyin Wang Y. Wang Jiwei Li Eduard H. Hovy Chen Guo 32 4 0 10 Oct 2024
ReIFE: Re-evaluating Instruction-Following Evaluation Yixin Liu Kejian Shi Alexander R. Fabbri Yilun Zhao Peifeng Wang Chien-Sheng Wu Shafiq Joty Arman Cohan 16 6 0 09 Oct 2024
TOWER: Tree Organized Weighting for Evaluating Complex Instructions Noah Ziems Zhihan Zhang Meng-Long Jiang ALM 21 0 0 08 Oct 2024
As Simple as Fine-tuning: LLM Alignment via Bidirectional Negative Feedback Loss Xin Mao Feng-Lin Li Huimin Xu Wei Zhang Wang Chen A. Luu 27 1 0 07 Oct 2024
TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generation Jonathan Cook Tim Rocktaschel Jakob Foerster Dennis Aumiller Alex Wang ALM 29 9 0 04 Oct 2024
Training Language Models to Win Debates with Self-Play Improves Judge Accuracy Samuel Arnesen David Rein Julian Michael ELM 28 3 0 25 Sep 2024
HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models Haoran Que Feiyu Duan Liqun He Yutao Mou Wangchunshu Zhou ... Ge Zhang Junran Peng Zhaoxiang Zhang Songyang Zhang Kai Chen LM&MA ELM VLM 43 11 0 24 Sep 2024
Aligning Language Models Using Follow-up Likelihood as Reward Signal Chen Zhang Dading Chong Feng Jiang Chengguang Tang Anningzhe Gao Guohua Tang Haizhou Li ALM 29 2 0 20 Sep 2024
Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation Jasper Dekoninck Maximilian Baader Martin Vechev ALM 87 0 0 01 Sep 2024
ConsistencyTrack: A Robust Multi-Object Tracker with a Generation Strategy of Consistency Model Lifan Jiang Zhihui Wang Siqi Yin Guangxiao Ma Peng Zhang Boxi Wu DiffM 51 0 0 28 Aug 2024
Jamba-1.5: Hybrid Transformer-Mamba Models at Scale Jamba Team Barak Lenz Alan Arazi Amir Bergman Avshalom Manevich ... Yehoshua Cohen Yonatan Belinkov Y. Globerson Yuval Peleg Levy Y. Shoham 29 26 0 22 Aug 2024
The Future of Open Human Feedback Shachar Don-Yehiya Ben Burtenshaw Ramon Fernandez Astudillo Cailean Osborne Mimansa Jaiswal ... Omri Abend Jennifer Ding Sara Hooker Hannah Rose Kirk Leshem Choshen VLM ALM 59 4 0 15 Aug 2024
EXAONE 3.0 7.8B Instruction Tuned Language Model LG AI Research : Soyoung An Kyunghoon Bae Eunbi Choi ... Boseong Seo Sihoon Yang Heuiyeen Yeen Kyungjae Yoo Hyeongu Yun ELM ALM 41 10 0 07 Aug 2024
Compare without Despair: Reliable Preference Evaluation with Generation Separability Sayan Ghosh Tejas Srinivasan Swabha Swayamdipta 29 2 0 02 Jul 2024
From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline Tianle Li Wei-Lin Chiang Evan Frick Lisa Dunlap Tianhao Wu Banghua Zhu Joseph E. Gonzalez Ion Stoica ALM 30 115 0 17 Jun 2024