Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text

14 February 2022

Papers citing "Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text"

49 / 49 papers shown

Title
SPHERE: An Evaluation Card for Human-AI Systems Qianou Ma Dora Zhao Xinran Zhao Chenglei Si Chenyang Yang Ryan Louie Ehud Reiter Diyi Yang Tongshuang Wu ALM 50 0 0 24 Mar 2025
A linguistically-motivated evaluation methodology for unraveling model's abilities in reading comprehension tasks Elie Antoine Frédéric Béchet Géraldine Damnati Philippe Langlais 49 1 0 29 Jan 2025
Learning to Explore and Select for Coverage-Conditioned Retrieval-Augmented Generation Takyoung Kim Kyungjae Lee Y. Jang Ji Yong Cho Gangwoo Kim Minseok Cho Moontae Lee 80 0 0 28 Jan 2025
Leveraging Entailment Judgements in Cross-Lingual Summarisation Huajian Zhang Laura Perez-Beltrachini HILM 29 0 0 01 Aug 2024
Beyond Metrics: A Critical Analysis of the Variability in Large Language Model Evaluation Frameworks Marco AF Pimentel Clément Christophe Tathagata Raha Prateek Munjal Praveen K Kanithi Shadab Khan ELM 24 2 0 29 Jul 2024
MCRanker: Generating Diverse Criteria On-the-Fly to Improve Point-wise LLM Rankers Fang Guo Wenyu Li Honglei Zhuang Yun Luo Yafu Li Qi Zhu Le Yan Yue Zhang ALM 63 6 0 18 Apr 2024
How Much Annotation is Needed to Compare Summarization Models? Chantal Shaib Joe Barrow Alexa F. Siu Byron C. Wallace A. Nenkova 29 2 0 28 Feb 2024
SaGE: Evaluating Moral Consistency in Large Language Models Vamshi Bonagiri Sreeram Vennam Priyanshul Govil Ponnurangam Kumaraguru Manas Gaur ELM 41 0 0 21 Feb 2024
Event-Keyed Summarization William Gantt Alexander Martin Pavlo Kuchmiichuk Aaron Steven White 9 1 0 10 Feb 2024
Mitigating Open-Vocabulary Caption Hallucinations Assaf Ben-Kish Moran Yanuka Morris Alper Raja Giryes Hadar Averbuch-Elor MLLM VLM 11 6 0 06 Dec 2023
X-Eval: Generalizable Multi-aspect Text Evaluation via Augmented Instruction Tuning with Auxiliary Evaluation Aspects Minqian Liu Ying Shen Zhiyang Xu Yixin Cao Eunah Cho Vaibhav Kumar Reza Ghanadan Lifu Huang ELM LM&MA ALM 36 25 0 15 Nov 2023
Llama 2: Open Foundation and Fine-Tuned Chat Models Hugo Touvron Louis Martin Kevin R. Stone Peter Albert Amjad Almahairi ... Sharan Narang Aurelien Rodriguez Robert Stojnic Sergey Edunov Thomas Scialom AI4MH ALM 70 10,890 0 18 Jul 2023
Mini-Giants: "Small" Language Models and Open Source Win-Win Zhengping Zhou Lezhi Li Xinxi Chen Andy Li SyDa ALM MoE 24 5 0 17 Jul 2023
DecompEval: Evaluating Generated Texts as Unsupervised Decomposed Question Answering Pei Ke Fei Huang Fei Mi Yasheng Wang Qun Liu Xiaoyan Zhu Minlie Huang ReLM ELM 29 10 0 13 Jul 2023
Rethinking Model Evaluation as Narrowing the Socio-Technical Gap Q. V. Liao Ziang Xiao ALM ELM 38 26 0 01 Jun 2023
A Critical Evaluation of Evaluations for Long-form Question Answering Fangyuan Xu Yixiao Song Mohit Iyyer Eunsol Choi ELM 17 94 0 29 May 2023
What are the Desired Characteristics of Calibration Sets? Identifying Correlates on Long Form Scientific Summarization Griffin Adams Bichlien H. Nguyen Jake A. Smith Yingce Xia Shufang Xie Anna Ostropolets Budhaditya Deb Yuan Chen Tristan Naumann Noémie Elhadad 13 8 0 12 May 2023
Large language models effectively leverage document-level context for literary translation, but critical errors persist Marzena Karpinska Mohit Iyyer 31 81 0 06 Apr 2023
BloombergGPT: A Large Language Model for Finance Shijie Wu Ozan Irsoy Steven Lu Vadim Dabravolski Mark Dredze Sebastian Gehrmann P. Kambadur David S. Rosenberg Gideon Mann AIFin 34 770 0 30 Mar 2023
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model BigScience Workshop : Teven Le Scao Angela Fan Christopher Akiki ... Zhongli Xie Zifan Ye M. Bras Younes Belkada Thomas Wolf VLM 27 2,297 0 09 Nov 2022
Dialect-robust Evaluation of Generated Text Jiao Sun Thibault Sellam Elizabeth Clark Tu Vu Timothy Dozat Dan Garrette Aditya Siddhant Jacob Eisenstein Sebastian Gehrmann 13 19 0 02 Nov 2022
Towards Interpretable Summary Evaluation via Allocation of Contextual Embeddings to Reference Text Topics Ben Schaper Christopher Lohse Marcell Streile Andrea Giovannini Richard Osuala 11 1 0 25 Oct 2022
DEMETR: Diagnosing Evaluation Metrics for Translation Marzena Karpinska N. Raj Katherine Thai Yixiao Song Ankita Gupta Mohit Iyyer 13 35 0 25 Oct 2022
REV: Information-Theoretic Evaluation of Free-Text Rationales Hanjie Chen Faeze Brahman Xiang Ren Yangfeng Ji Yejin Choi Swabha Swayamdipta 84 22 0 10 Oct 2022
RankGen: Improving Text Generation with Large Ranking Models Kalpesh Krishna Yapei Chang John Wieting Mohit Iyyer AIMat 11 68 0 19 May 2022
Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implications Kaitlyn Zhou Su Lin Blodgett Adam Trischler Hal Daumé Kaheer Suleman Alexandra Olteanu ELM 94 25 0 13 May 2022
Adaptive Sampling Strategies to Construct Equitable Training Datasets William Cai R. Encarnación Bobbie Chern S. Corbett-Davies Miranda Bogen Stevie Bergman Sharad Goel 77 29 0 31 Jan 2022
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason W. Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter F. Xia Ed H. Chi Quoc Le Denny Zhou LM&Ro LRM AI4CE ReLM 315 8,261 0 28 Jan 2022
Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand Jungo Kasai Keisuke Sakaguchi Ronan Le Bras Lavinia Dunagan Jacob Morrison Alexander R. Fabbri Yejin Choi Noah A. Smith 49 39 0 08 Dec 2021
NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation Kaustubh D. Dhole Varun Gangal Sebastian Gehrmann Aadesh Gupta Zhenhao Li ... Tianbao Xie Usama Yaseen Michael A. Yee Jing Zhang Yue Zhang 159 86 0 06 Dec 2021
A Survey of NLP-Related Crowdsourcing HITs: what works and what does not Jessica Huynh Jeffrey P. Bigham M. Eskénazi 46 18 0 09 Nov 2021
A Framework for Deprecating Datasets: Standardizing Documentation, Identification, and Communication A. Luccioni Frances Corry H. Sridharan Mike Ananny J. Schultz Kate Crawford 36 28 0 18 Oct 2021
Finding a Balanced Degree of Automation for Summary Evaluation Shiyue Zhang Mohit Bansal 47 43 0 23 Sep 2021
The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation Marzena Karpinska Nader Akoury Mohit Iyyer 204 106 0 14 Sep 2021
Just What do You Think You're Doing, Dave?' A Checklist for Responsible Data Use in NLP Anna Rogers Timothy Baldwin Kobi Leins 102 64 0 14 Sep 2021
Perturbation CheckLists for Evaluating NLG Evaluation Metrics Ananya B. Sai Tanay Dixit D. Y. Sheth S. Mohan Mitesh M. Khapra AAML 94 55 0 13 Sep 2021
Does It Capture STEL? A Modular, Similarity-based Linguistic Style Evaluation Framework Anna Wegmann D. Nguyen 24 12 0 10 Sep 2021
Deduplicating Training Data Makes Language Models Better Katherine Lee Daphne Ippolito A. Nystrom Chiyuan Zhang Douglas Eck Chris Callison-Burch Nicholas Carlini SyDa 237 588 0 14 Jul 2021
Evaluating Attribution in Dialogue Systems: The BEGIN Benchmark Nouha Dziri Hannah Rashkin Tal Linzen David Reitter ALM 185 79 0 30 Apr 2021
Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics Artidoro Pagnoni Vidhisha Balachandran Yulia Tsvetkov HILM 215 305 0 27 Apr 2021
Beyond Fair Pay: Ethical Implications of NLP Crowdsourcing Boaz Shmueli Jan Fell Soumya Ray Lun-Wei Ku 100 86 0 20 Apr 2021
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics Sebastian Gehrmann Tosin P. Adewumi Karmanya Aggarwal Pawan Sasanka Ammanamanchi Aremu Anuoluwapo ... Nishant Subramani Wei-ping Xu Diyi Yang Akhila Yerukola Jiawei Zhou VLM 238 284 0 02 Feb 2021
Robustness Gym: Unifying the NLP Evaluation Landscape Karan Goel Nazneen Rajani Jesse Vig Samson Tan Jason M. Wu Stephan Zheng Caiming Xiong Mohit Bansal Christopher Ré AAML OffRL OOD 138 136 0 13 Jan 2021
GO FIGURE: A Meta Evaluation of Factuality in Summarization Saadia Gabriel Asli Celikyilmaz Rahul Jha Yejin Choi Jianfeng Gao HILM 227 96 0 24 Oct 2020
Factual Error Correction for Abstractive Summarization Models Mengyao Cao Yue Dong Jiapeng Wu Jackie C.K. Cheung HILM KELM 167 159 0 17 Oct 2020
With Little Power Comes Great Responsibility Dallas Card Peter Henderson Urvashi Khandelwal Robin Jia Kyle Mahowald Dan Jurafsky 225 115 0 13 Oct 2020
Towards A Rigorous Science of Interpretable Machine Learning Finale Doshi-Velez Been Kim XAI FaML 225 3,658 0 28 Feb 2017
SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents Ramesh Nallapati Feifei Zhai Bowen Zhou 200 1,249 0 14 Nov 2016
Teaching Machines to Read and Comprehend Karl Moritz Hermann Tomás Kociský Edward Grefenstette L. Espeholt W. Kay Mustafa Suleyman Phil Blunsom 170 3,504 0 10 Jun 2015