Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations

12 October 2021

Papers citing "Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations"

50 / 147 papers shown

Title
Hateful Person or Hateful Model? Investigating the Role of Personas in Hate Speech Detection by Large Language Models Shuzhou Yuan Ercong Nie Mario Tawfelis Helmut Schmid Hinrich Schütze Michael Färber 18 0 0 10 Jun 2025
HESEIA: A community-based dataset for evaluating social biases in large language models, co-designed in real school settings in Latin America Guido Ivetta Marcos J. Gomez Sofía Martinelli Pietro Palombini M. Emilia Echeveste Nair Carolina Mazzeo Beatriz Busaniche Luciana Benotti VLM 22 0 0 30 May 2025
Large Language Models Do Multi-Label Classification Differently Marcus Ma Georgios Chochlakis Niyantha Maruthu Pandiyan Jesse Thomason Shrikanth Narayanan 98 1 0 23 May 2025
Meta-PerSER: Few-Shot Listener Personalized Speech Emotion Recognition via Meta-learning Liang-Yeh Shen Shi-Xin Fang Yi-Cheng Lin Huang-Cheng Chou Hung-yi Lee 36 0 0 22 May 2025
Humans Hallucinate Too: Language Models Identify and Correct Subjective Annotation Errors With Label-in-a-Haystack Prompts Georgios Chochlakis Peter Wu Arjun Bedi Marcus Ma Kristina Lerman Shrikanth Narayanan 182 0 0 22 May 2025
Reliable Decision Support with LLMs: A Framework for Evaluating Consistency in Binary Text Classification Applications Fadel M. Megahed Ying-Ju Chen L. Allision Jones-Farmer Younghwa Lee Jiawei Brooke Wang Inez M. Zwetsloot 65 0 0 20 May 2025
Conflicts in Texts: Data, Implications and Challenges Siyi Liu Dan Roth 443 0 0 28 Apr 2025
Towards a comprehensive taxonomy of online abusive language informed by machine leaning Samaneh Hosseini Moghaddam Kelly Lyons Cheryl Regehr Vivek Goel Kaitlyn Regehr 60 0 0 24 Apr 2025
Graphically Speaking: Unmasking Abuse in Social Media with Conversation Insights Célia Nouri Jean-Philippe Cointet Chloé Clavel 82 0 0 02 Apr 2025
Evaluating how LLM annotations represent diverse views on contentious topics Megan A. Brown Shubham Atreja Libby Hemphill Patrick Y. Wu 425 0 0 29 Mar 2025
QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks? Belinda Z. Li Been Kim Zehao Wang LRM 101 6 0 28 Mar 2025
The Case for "Thick Evaluations" of Cultural Representation in AI Rida Qadri Mark Díaz Ding Wang Michael Madaio 91 4 0 24 Mar 2025
CULEMO: Cultural Lenses on Emotion -- Benchmarking LLMs for Cross-Cultural Emotion Understanding Tadesse Destaw Belay Ahmed Haj Ahmed Alvin Grissom II Iqra Ameer Grigori Sidorov Olga Kolesnikova Seid Muhie Yimam 151 2 0 12 Mar 2025
Embracing Diversity: A Multi-Perspective Approach with Soft Labels Benedetta Muscato Praveen Bushipaka Gizem Gezici Lucia Passaro F. Giannotti Tommaso Cucinotta 105 0 0 01 Mar 2025
Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals' Subjective Text Perceptions Matthias Orlikowski Jiaxin Pei Paul Röttger Philipp Cimiano David Jurgens Dirk Hovy 135 3 0 28 Feb 2025
From Deception to Perception: The Surprising Benefits of Deepfakes for Detecting, Measuring, and Mitigating Bias Yizhi Liu Balaji Padmanabhan Siva Viswanathan 76 0 0 16 Feb 2025
RideKE: Leveraging Low-Resource, User-Generated Twitter Content for Sentiment and Emotion Detection in Kenyan Code-Switched Dataset Naome A. Etori Maria Gini 169 3 0 10 Feb 2025
AI Alignment at Your Discretion Maarten Buyl Hadi Khalaf C. M. Verdun Lucas Monteiro Paes Caio Vieira Machado Flavio du Pin Calmon 114 1 0 10 Feb 2025
FuocChuVIP123 at CoMeDi Shared Task: Disagreement Ranking with XLM-Roberta Sentence Embeddings and Deep Neural Regression Phuoc Duong Huy Chu 69 1 0 21 Jan 2025
Beyond Dataset Creation: Critical View of Annotation Variation and Bias Probing of a Dataset for Online Radical Content Detection Arij Riabi Virginie Mouilleron Menel Mahamdi Wissam Antoun Djamé Seddah 131 1 0 16 Dec 2024
Exploring the Influence of Label Aggregation on Minority Voices: Implications for Dataset Bias and Model Training Mugdha Pandya Nafise Sadat Moosavi Diana Maynard 128 1 0 05 Dec 2024
AI-EDI-SPACE: A Co-designed Dataset for Evaluating the Quality of Public Spaces Shreeyash Gowaikar Hugo Berard Rashid Mushkani Emmanuel Beaudry Marchand Toumadher Ammar Shin Koseki 65 1 0 01 Nov 2024
What Makes An Expert? Reviewing How ML Researchers Define "Expert" Mark Díaz Angela D. R. Smith 54 2 0 31 Oct 2024
Insights on Disagreement Patterns in Multimodal Safety Perception across Diverse Rater Groups Charvi Rastogi Tian Huey Teh Pushkar Mishra Roma Patel Zoe C. Ashwood ... Alicia Parrish Ding Wang Vinodkumar Prabhakaran Lora Aroyo Verena Rieser EGVM 54 2 0 22 Oct 2024
ComPO: Community Preferences for Language Model Personalization Sachin Kumar Chan Young Park Yulia Tsvetkov Noah A. Smith Hannaneh Hajishirzi 78 8 0 21 Oct 2024
Reducing annotator bias by belief elicitation Terne Sasha Thorn Jakobsen Andreas Bjerre-Nielsen Robert Böhm 75 0 0 21 Oct 2024
Mitigating Biases to Embrace Diversity: A Comprehensive Annotation Benchmark for Toxic Language Xinmeng Hou 68 1 0 17 Oct 2024
Aggregation Artifacts in Subjective Tasks Collapse Large Language Models' Posteriors Georgios Chochlakis Alexandros Potamianos Kristina Lerman Shrikanth Narayanan 151 2 0 17 Oct 2024
Accurate and Data-Efficient Toxicity Prediction when Annotators Disagree Harbani Jaggi Kashyap Murali Eve Fleisig Erdem Bıyık 40 1 0 16 Oct 2024
Re-examining Sexism and Misogyny Classification with Annotator Attitudes Aiqi Jiang Nikolas Vitsakis Tanvi Dinkar Gavin Abercrombie Ioannis Konstas 110 2 0 04 Oct 2024
ARTICLE: Annotator Reliability Through In-Context Learning Sujan Dutta Deepak Pandita Tharindu Cyril Weerasooriya Marcos Zampieri Christopher M. Homan Ashiqur R. KhudaBukhsh 59 0 0 18 Sep 2024
Performance of Human Annotators in Object Detection and Segmentation of Remotely Sensed Data Roni Blushtein-Livnon T. Svoray Michael Dorman 103 2 0 16 Sep 2024
Keeping Humans in the Loop: Human-Centered Automated Annotation with Generative AI Nicholas Pangakis Samuel Wolken 69 4 0 14 Sep 2024
Beyond Preferences in AI Alignment Tan Zhi-Xuan Micah Carroll Matija Franklin Hal Ashton 138 18 0 30 Aug 2024
Crowd-Calibrator: Can Annotator Disagreement Inform Calibration in Subjective Tasks? Urja Khurana Eric T. Nalisnick Antske Fokkens Swabha Swayamdipta 107 4 0 26 Aug 2024
Uncovering Biases with Reflective Large Language Models Edward Y. Chang 28 0 0 24 Aug 2024
The Whole Is Bigger Than the Sum of Its Parts: Modeling Individual Annotators to Capture Emotional Variability James Tavernor Yara S. El-Tawil E. Provost 52 1 0 21 Aug 2024
A Theory-Based Explainable Deep Learning Architecture for Music Emotion H. Fong Vineet Kumar K. Sudhir FAtt 19 2 0 13 Aug 2024
PEFT-U: Parameter-Efficient Fine-Tuning for User Personalization Christopher Clarke Yuzhao Heng Lingjia Tang Jason Mars 55 4 0 25 Jul 2024
Voices in a Crowd: Searching for Clusters of Unique Perspectives Nikolas Vitsakis Amit Parekh Ioannis Konstas 80 1 0 19 Jul 2024
GPT Assisted Annotation of Rhetorical and Linguistic Features for Interpretable Propaganda Technique Detection in News Text Kyle Hamilton Luca Longo Bojan Bozic 74 1 0 16 Jul 2024
Position: Measure Dataset Diversity, Don't Just Claim It Dora Zhao Jerone T. A. Andrews Orestis Papakyriakopoulos Alice Xiang 108 20 0 11 Jul 2024
NADI 2024: The Fifth Nuanced Arabic Dialect Identification Shared Task Muhammad Abdul-Mageed Amr Keleg AbdelRahim Elmadany Chiyu Zhang Injy Hamed Walid Magdy Houda Bouamor Nizar Habash 87 19 0 06 Jul 2024
"Seeing the Big through the Small": Can LLMs Approximate Human Judgment Distributions on NLI from a Few Explanations? Beiduo Chen Xinpeng Wang Siyao Peng Robert Litschko Anna Korhonen Barbara Plank 115 9 0 25 Jun 2024
A multitask learning framework for leveraging subjectivity of annotators to identify misogyny Jason Angel S. Aroyehun Grigori Sidorov Alexander Gelbukh 28 0 0 22 Jun 2024
Let Guidelines Guide You: A Prescriptive Guideline-Centered Data Annotation Methodology Federico Ruggeri Eleonora Misino Arianna Muti Katerina Korre Paolo Torroni Alberto Barrón-Cedeño 103 1 0 20 Jun 2024
Extrinsic Evaluation of Cultural Competence in Large Language Models Shaily Bhatt Fernando Diaz ELM EGVM 110 9 0 17 Jun 2024
Survey for Landing Generative AI in Social and E-commerce Recsys -- the Industry Perspectives Da Xu Danqing Zhang Guangyu Yang Bo Yang Shuyuan Xu Lingling Zheng Cindy Liang 36 3 0 10 Jun 2024
A Taxonomy of Challenges to Curating Fair Datasets Dora Zhao M. Scheuerman Pooja Chitre Jerone T. A. Andrews Georgia Panagiotidou Shawn Walker Kathleen H. Pine Alice Xiang 97 2 0 10 Jun 2024
Whose Preferences? Differences in Fairness Preferences and Their Impact on the Fairness of AI Utilizing Human Feedback Emilia Agis Lerner Florian E. Dorner Elliott Ash Naman Goel 59 1 0 09 Jun 2024