Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations

12 October 2021

Papers citing "Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations"

47 / 147 papers shown

Title
''Fifty Shades of Bias'': Normative Ratings of Gender Bias in GPT Generated English Text Rishav Hada Agrima Seth Harshita Diddee Kalika Bali 97 17 0 26 Oct 2023
"One-Size-Fits-All"? Examining Expectations around What Constitute "Fair" or "Good" NLG System Behaviors Li Lucy Su Lin Blodgett Milad Shokouhi Hanna M. Wallach Alexandra Olteanu 91 9 0 23 Oct 2023
ACTOR: Active Learning with Annotator-specific Classification Heads to Embrace Human Label Variation Xinpeng Wang Barbara Plank 68 7 0 23 Oct 2023
Unveiling the Multi-Annotation Process: Examining the Influence of Annotation Quantity and Instance Difficulty on Model Performance Pritam Kadasi Mayank Singh 57 3 0 23 Oct 2023
Which Prompts Make The Difference? Data Prioritization For Efficient Human LLM Evaluation M. Boubdir Edward Kim Beyza Ermis Marzieh Fadaee Sara Hooker ALM 85 19 0 22 Oct 2023
Eliciting Model Steering Interactions from Users via Data and Visual Design Probes Anamaria Crisan Maddie Shang Eric Brochu 70 3 0 12 Oct 2023
The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values Hannah Rose Kirk Andrew M. Bean Bertie Vidgen Paul Röttger Scott A. Hale ALM 113 50 0 11 Oct 2023
The Empty Signifier Problem: Towards Clearer Paradigms for Operationalising "Alignment" in Large Language Models Hannah Rose Kirk Bertie Vidgen Paul Röttger Scott A. Hale 112 4 0 03 Oct 2023
It HAS to be Subjective: Human Annotator Simulation via Zero-shot Density Estimation Wen Wu Jiajun He Chuxu Zhang P. Woodland 55 1 0 30 Sep 2023
PopBERT. Detecting populism and its host ideologies in the German Bundestag Lukas Erhard Sara Hanke Uwe Remer A. Falenska R. Heiberger 63 2 0 22 Sep 2023
On the Challenges of Building Datasets for Hate Speech Detection Vitthal Bhandari 67 1 0 06 Sep 2023
Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Duties Taylor Sorensen Liwei Jiang Jena D. Hwang Sydney Levine Valentina Pyatkin ... Kavel Rao Chandra Bhagavatula Maarten Sap J. Tasioulas Yejin Choi SLR 119 60 0 02 Sep 2023
BatchPrompt: Accomplish more with less Jianzhe Lin Maurice Diesendruck Liang Du Robin Abraham LRM 96 10 0 01 Sep 2023
Exploring Cross-Cultural Differences in English Hate Speech Annotations: From Dataset Construction to Analysis Nayeon Lee Chani Jung Jun-Hee Myung Jiho Jin Jose Camacho-Collados Juho Kim Alice Oh 98 23 0 31 Aug 2023
Large Language Models and Knowledge Graphs: Opportunities and Challenges Jeff Z. Pan Simon Razniewski Jan-Christoph Kalo Sneha Singhania Jiaoyan Chen ... Gerard de Melo A. Bonifati Edlira Vakaj M. Dragoni D. Graux KELM 91 78 0 11 Aug 2023
Designing Closed-Loop Models for Task Allocation Vijay Keswani L. E. Celis K. Kenthapadi Matthew Lease 59 0 0 31 May 2023
Consensus and Subjectivity of Skin Tone Annotation for ML Fairness Candice Schumann Gbolahan O. Olanubi Auriel Wright Ellis P. Monk Courtney Heldreth Susanna Ricco 110 24 0 16 May 2023
When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks Eve Fleisig Rediet Abebe Dan Klein 131 49 0 11 May 2023
iLab at SemEval-2023 Task 11 Le-Wi-Di: Modelling Disagreement or Modelling Perspectives? Nikolas Vitsakis Amit Parekh Tanvi Dinkar Gavin Abercrombie Ioannis Konstas Verena Rieser 98 10 0 10 May 2023
SemEval-2023 Task 10: Explainable Detection of Online Sexism Hannah Rose Kirk Wenjie Yin Bertie Vidgen Paul Röttger 81 122 0 07 Mar 2023
Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Information Ruyuan Wan Jaehyung Kim Dongyeop Kang 55 38 0 12 Jan 2023
AnnoBERT: Effectively Representing Multiple Annotators' Label Choices to Improve Hate Speech Detection Wenjie Yin Vibhor Agarwal Aiqi Jiang A. Zubiaga Nishanth R. Sastry 91 15 0 20 Dec 2022
Human-Guided Fair Classification for Natural Language Processing Florian E.Dorner Momchil Peychev Nikola Konstantinov Naman Goel Elliott Ash Martin Vechev FaML 71 4 0 20 Dec 2022
SeedBERT: Recovering Annotator Rating Distributions from an Aggregated Label A. Sampath Victoria Lin Louis-Philippe Morency 29 2 0 23 Nov 2022
"It's Not Just Hate'': A Multi-Dimensional Perspective on Detecting Harmful Speech Online Federico Bianchi S. A. Hills Patrícia G. C. Rossini Dirk Hovy Rebekah Tromble N. Tintarev 95 15 0 28 Oct 2022
Unifying Data Perspectivism and Personalization: An Application to Social Norms Joan Plepi Béla Neuendorf Lucie Flek Charles F Welch 117 21 0 26 Oct 2022
Trustworthy Human Computation: A Survey H. Kashima S. Oyama Hiromi Arai Junichiro Mori 82 1 0 22 Oct 2022
Noise Audits Improve Moral Foundation Classification Negar Mokhberian F. R. Hopp Bahareh Harandizadeh Fred Morstatter Kristina Lerman NoLa 82 7 0 13 Oct 2022
Explainable Abuse Detection as Intent Classification and Slot Filling Agostina Calabrese Bjorn Ross Mirella Lapata 93 11 0 06 Oct 2022
Investigating Reasons for Disagreement in Natural Language Inference Nan-Jiang Jiang M. Marneffe 72 27 0 07 Sep 2022
A Holistic Approach to Undesired Content Detection in the Real World Todor Markov Chong Zhang Sandhini Agarwal Tyna Eloundou Teddy Lee Steven Adler Angela Jiang L. Weng 125 237 0 05 Aug 2022
More Data Can Lead Us Astray: Active Data Acquisition in the Presence of Label Bias Yunyi Li Maria De-Arteaga M. Saar-Tsechansky FaML 83 3 0 15 Jul 2022
Is one annotation enough? A data-centric image classification benchmark for noisy and ambiguous label estimation Lars Schmarje Vasco Grossmann Claudius Zelenka S. Dippel R. Kiko ... M. Pastell J. Stracke A. Valros N. Volkmann Reinahrd Koch 115 37 0 13 Jul 2022
Empathic Conversations: A Multi-level Dataset of Contextualized Conversations Damilola Omitaomu Shabnam Tafreshi Tingting Liu Sven Buechel Chris Callison-Burch J. Eichstaedt Lyle Ungar João Sedoc 104 50 0 25 May 2022
Evaluation Gaps in Machine Learning Practice Ben Hutchinson Negar Rostamzadeh Christina Greer Katherine A. Heller Vinodkumar Prabhakaran ELM 98 63 0 11 May 2022
Justice in Misinformation Detection Systems: An Analysis of Algorithms, Stakeholders, and Potential Harms Terrence Neumann Maria De-Arteaga S. Fazelpour 97 26 0 28 Apr 2022
Challenges and Strategies in Cross-Cultural NLP Daniel Hershcovich Stella Frank Heather Lent Miryam de Lhoneux Mostafa Abdou ... Ruixiang Cui Constanza Fierro Katerina Margatina Phillip Rust Anders Søgaard 130 182 0 18 Mar 2022
ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection Thomas Hartvigsen Saadia Gabriel Hamid Palangi Maarten Sap Dipankar Ray Ece Kamar 92 391 0 17 Mar 2022
The Ghost in the Machine has an American accent: value conflict in GPT-3 Rebecca Lynn Johnson Giada Pistilli Natalia Menédez-González Leslye Denisse Dias Duran Enrico Panai Julija Kalpokienė D. Bertulfo 104 90 0 15 Mar 2022
The worst of both worlds: A comparative analysis of errors in learning from data in psychology and machine learning Jessica Hullman Sayash Kapoor Priyanka Nanayakkara Andrew Gelman Arvind Narayanan 147 39 0 12 Mar 2022
Toward More Meaningful Resources for Lower-resourced Languages Constantine Lignos Nolan Holley Chester Palen-Michel Jonne Saleva 56 20 0 24 Feb 2022
Jury Learning: Integrating Dissenting Voices into Machine Learning Models Mitchell L. Gordon Michelle S. Lam J. Park Kayur Patel Jeffrey T. Hancock Tatsunori Hashimoto Michael S. Bernstein 87 154 0 07 Feb 2022
NaijaSenti: A Nigerian Twitter Sentiment Corpus for Multilingual Sentiment Analysis Shamsuddeen Hassan Muhammad David Ifeoluwa Adelani Sebastian Ruder Ibrahim Said Ahmad Idris Abdulmumin ... Chris C. Emezue Saheed Abdul Anuoluwapo Aremu Alipio Jeorge P. Brazdil 83 100 0 20 Jan 2022
Ground-Truth, Whose Truth? -- Examining the Challenges with Annotating Toxic Text Datasets Kofi Arhin Ioana Baldini Dennis L. Wei Karthikeyan N. Ramamurthy Moninder Singh 61 20 0 07 Dec 2021
On Releasing Annotator-Level Labels and Information in Datasets Vinodkumar Prabhakaran Aida Mostafazadeh Davani Mark Díaz 109 150 0 12 Oct 2021
Diversity in Sociotechnical Machine Learning Systems S. Fazelpour Maria De-Arteaga 88 38 0 19 Jul 2021
Leveraging Expert Consistency to Improve Algorithmic Decision Support Maria De-Arteaga Vincent Jeanselme A. Dubrawski Alexandra Chouldechova 72 22 0 24 Jan 2021