On Releasing Annotator-Level Labels and Information in Datasets

12 October 2021

Papers citing "On Releasing Annotator-Level Labels and Information in Datasets"

24 / 24 papers shown

Title
AI Alignment at Your Discretion Maarten Buyl Hadi Khalaf C. M. Verdun Lucas Monteiro Paes Caio Vieira Machado Flavio du Pin Calmon 42 0 0 10 Feb 2025
Correcting Annotator Bias in Training Data: Population-Aligned Instance Replication (PAIR) Stephanie Eckman Bolei Ma Christoph Kern Rob Chew Barbara Plank Frauke Kreuter 41 0 0 12 Jan 2025
Aggregation Artifacts in Subjective Tasks Collapse Large Language Models' Posteriors Georgios Chochlakis Alexandros Potamianos Kristina Lerman Shrikanth Narayanan 30 0 0 17 Oct 2024
Human and LLM Biases in Hate Speech Annotations: A Socio-Demographic Analysis of Annotators and Targets Tommaso Giorgi Lorenzo Cima T. Fagni M. Avvenuti S. Cresci 40 9 0 10 Oct 2024
Bayesian WeakS-to-Strong from Text Classification to Generation Ziyun Cui Ziyang Zhang Wen Wu Wen Wu Chao Zhang 31 1 0 24 May 2024
Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback Vincent Conitzer Rachel Freedman J. Heitzig Wesley H. Holliday Bob M. Jacobs ... Eric Pacuit Stuart Russell Hailey Schoelkopf Emanuel Tewolde W. Zwicker 33 28 0 16 Apr 2024
Cost-Efficient Subjective Task Annotation and Modeling through Few-Shot Annotator Adaptation Preni Golazizian Ali Omrani Alireza S. Ziabari Morteza Dehghani 18 1 0 21 Feb 2024
GRASP: A Disagreement Analysis Framework to Assess Group Associations in Perspectives Vinodkumar Prabhakaran Christopher Homan Lora Aroyo Aida Mostafazadeh Davani Alicia Parrish Alex S. Taylor Mark Díaz Ding Wang Greg Serapio-García 34 9 0 09 Nov 2023
Semantic and Expressive Variation in Image Captions Across Languages Andre Ye Sebastin Santy Jena D. Hwang Amy X. Zhang Ranjay Krishna VLM 48 3 0 22 Oct 2023
Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Prompting Tilman Beck Hendrik Schuff Anne Lauscher Iryna Gurevych 35 32 0 13 Sep 2023
The Ecological Fallacy in Annotation: Modelling Human Label Variation goes beyond Sociodemographics Matthias Orlikowski Paul Röttger Philipp Cimiano Italy 26 26 0 20 Jun 2023
Intersectionality in Conversational AI Safety: How Bayesian Multilevel Models Help Understand Diverse Perceptions of Safety Christopher Homan Greg Serapio-García Lora Aroyo Mark Díaz Alicia Parrish Vinodkumar Prabhakaran Alex S. Taylor Ding Wang 22 9 0 20 Jun 2023
What's the Meaning of Superhuman Performance in Today's NLU? Simone Tedeschi Johan Bos T. Declerck Jan Hajic Daniel Hershcovich ... Simon Krek Steven Schockaert Rico Sennrich Ekaterina Shutova Roberto Navigli ELM LM&MA VLM ReLM LRM 34 26 0 15 May 2023
When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks Eve Fleisig Rediet Abebe Dan Klein 21 42 0 11 May 2023
Design Choices for Crowdsourcing Implicit Discourse Relations: Revealing the Biases Introduced by Task Design Valentina Pyatkin Frances Yung Merel C. J. Scholman Reut Tsarfaty Ido Dagan Vera Demberg 19 12 0 03 Apr 2023
SemEval-2023 Task 10: Explainable Detection of Online Sexism Hannah Rose Kirk Wenjie Yin Bertie Vidgen Paul Röttger 10 117 0 07 Mar 2023
The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation Barbara Plank 30 96 0 04 Nov 2022
Stop Measuring Calibration When Humans Disagree Joris Baan Wilker Aziz Barbara Plank Raquel Fernández 24 53 0 28 Oct 2022
A Keyword Based Approach to Understanding the Overpenalization of Marginalized Groups by English Marginal Abuse Models on Twitter Kyra Yee Alice Schoenauer Sebag Olivia Redfield Emily Sheng Matthias Eck Luca Belli 20 2 0 07 Oct 2022
Eliciting and Learning with Soft Labels from Every Annotator K. M. Collins Umang Bhatt Adrian Weller 11 44 0 02 Jul 2022
From Stance to Concern: Adaptation of Propositional Analysis to New Tasks and Domains Brodie Mather Bonnie J. Dorr Adam Dalton William de Beaumont Owen Rambow Sonja M. Schmer-Galunder 22 8 0 20 Mar 2022
Challenges and Strategies in Cross-Cultural NLP Daniel Hershcovich Stella Frank Heather Lent Miryam de Lhoneux Mostafa Abdou ... Ruixiang Cui Constanza Fierro Katerina Margatina Phillip Rust Anders Søgaard 41 162 0 18 Mar 2022
Jury Learning: Integrating Dissenting Voices into Machine Learning Models Mitchell L. Gordon Michelle S. Lam J. Park Kayur Patel Jeffrey T. Hancock Tatsunori Hashimoto Michael S. Bernstein 19 146 0 07 Feb 2022
Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection Maarten Sap Swabha Swayamdipta Laura Vianna Xuhui Zhou Yejin Choi Noah A. Smith 29 266 0 15 Nov 2021