Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations

12 October 2021

Papers citing "Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations"

50 / 147 papers shown

Title
Explainability and Hate Speech: Structured Explanations Make Social Media Moderators Faster Agostina Calabrese Leonardo Neves Neil Shah Maarten W. Bos Björn Ross Mirella Lapata Francesco Barbieri FAtt 69 5 0 06 Jun 2024
Enhancing Zero-shot Text-to-Speech Synthesis with Human Feedback Chen Chen Yuchen Hu Wen Wu Helin Wang Chng Eng Siong Chao Zhang 93 12 0 02 Jun 2024
Hostile Counterspeech Drives Users From Hate Subreddits Daniel Hickey Matheus Schmitz D. Fessler P. Smaldino Kristina Lerman Goran Murić Keith Burghardt AAML 26 0 0 28 May 2024
Trust and Terror: Hazards in Text Reveal Negatively Biased Credulity and Partisan Negativity Bias Keith Burghardt D. Fessler Chyna Tang Anne C. Pisor Kristina Lerman 96 0 0 28 May 2024
Bayesian WeakS-to-Strong from Text Classification to Generation Ziyun Cui Ziyang Zhang Wen Wu Wen Wu Chao Zhang 127 3 0 24 May 2024
Exploring Subjectivity for more Human-Centric Assessment of Social Biases in Large Language Models Paula Akemi Aoyagui Sharon Ferguson Anastasia Kuzminykh 81 0 0 17 May 2024
Akal Badi ya Bias: An Exploratory Study of Gender Bias in Hindi Language Technology Rishav Hada Safiya Husain Varun Gumma Harshita Diddee Aditya Yadavalli ... Nidhi Kulkarni U. Gadiraju Aditya Vashistha Vivek Seshadri Kalika Bali 91 10 0 10 May 2024
Towards Geographic Inclusion in the Evaluation of Text-to-Image Models Melissa Hall Samuel J. Bell Candace Ross Adina Williams M. Drozdzal Adriana Romero Soriano EGVM 66 5 0 07 May 2024
Active Learning with Task Adaptation Pre-training for Speech Emotion Recognition Dongyuan Li Ying Zhang Yusong Wang Funakoshi Kataro Manabu Okumura 74 2 0 01 May 2024
Annotator-Centric Active Learning for Subjective NLP Tasks Michiel van der Meer Neele Falk P. Murukannaiah Enrico Liscio 96 9 0 24 Apr 2024
Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences Shreya Shankar J.D. Zamfirescu-Pereira Bjorn Hartmann Aditya G. Parameswaran Ian Arawjo ALM 93 97 0 18 Apr 2024
D3CODE: Disentangling Disagreements in Data across Cultures on Offensiveness Detection and Evaluation Aida Mostafazadeh Davani Mark Díaz Dylan K. Baker Vinodkumar Prabhakaran 74 10 0 16 Apr 2024
If there's a Trigger Warning, then where's the Trigger? Investigating Trigger Warnings at the Passage Level Matti Wiegmann Jennifer Rakete Magdalena Wolska Benno Stein Martin Potthast LLMSV 74 0 0 15 Apr 2024
Corpus Considerations for Annotator Modeling and Scaling O. O. Sarumi Béla Neuendorf Joan Plepi Lucie Flek Jorg Schlotterer Charles F Welch 130 1 0 02 Apr 2024
Can Humans Identify Domains? Maria Barrett Max Müller-Eberstein Elisa Bassignana Amalie Brogaard Pauli Mike Zhang Rob van der Goot 104 1 0 02 Apr 2024
NLP Systems That Can't Tell Use from Mention Censor Counterspeech, but Teaching the Distinction Helps Kristina Gligorić Myra Cheng Lucia Zheng Esin Durmus Dan Jurafsky 83 9 0 02 Apr 2024
Detectors for Safe and Reliable LLMs: Implementations, Uses, and Limitations Swapnaja Achintalwar Adriana Alvarado Garcia Ateret Anaby-Tavor Ioana Baldini Sara E. Berger ... Aashka Trivedi Kush R. Varshney Dennis L. Wei Shalisha Witherspooon Marcel Zalmanovici 87 11 0 09 Mar 2024
SeeGULL Multilingual: a Dataset of Geo-Culturally Situated Stereotypes Mukul Bhutani Kevin Robinson Vinodkumar Prabhakaran Shachi Dave Sunipa Dev 124 12 0 08 Mar 2024
Don't Blame the Data, Blame the Model: Understanding Noise and Bias When Learning from Subjective Annotations Abhishek Anand Negar Mokhberian Prathyusha Naresh Kumar Anweasha Saha Zihao He Ashwin Rao Fred Morstatter Kristina Lerman 74 7 0 06 Mar 2024
VariErr NLI: Separating Annotation Error from Human Label Variation Leon Weber-Genzel Siyao Peng M. Marneffe Barbara Plank 75 11 0 04 Mar 2024
Emotion Analysis in NLP: Trends, Gaps and Roadmap for Future Directions Flor Miriam Plaza del Arco Alba Curry Amanda Cercas Curry Dirk Hovy 100 18 0 02 Mar 2024
Surveying the Dead Minds: Historical-Psychological Text Analysis with Contextualized Construct Representation (CCR) for Classical Chinese Yuqi Chen Sixuan Li Ying Li Mohammad Atari 107 5 0 01 Mar 2024
Interpreting Predictive Probabilities: Model Confidence or Human Label Variation? Joris Baan Raquel Fernández Barbara Plank Wilker Aziz 112 11 0 25 Feb 2024
Farsight: Fostering Responsible AI Awareness During AI Application Prototyping Zijie J. Wang Chinmay Kulkarni Lauren Wilcox Michael Terry Michael A. Madaio 75 50 0 23 Feb 2024
Cost-Efficient Subjective Task Annotation and Modeling through Few-Shot Annotator Adaptation Preni Golazizian Ali Omrani Alireza S. Ziabari Morteza Dehghani 47 1 0 21 Feb 2024
KorNAT: LLM Alignment Benchmark for Korean Social Values and Common Knowledge Jiyoung Lee Minwoo Kim Seungho Kim Junghwan Kim Seunghyun Won Hwaran Lee Edward Choi ALM 127 17 0 21 Feb 2024
Quantifying the Persona Effect in LLM Simulations Tiancheng Hu Nigel Collier 62 70 0 16 Feb 2024
Discipline and Label: A WEIRD Genealogy and Social Theory of Data Annotation Andrew Smart Ding Wang Ellis Monk Mark Díaz Atoosa Kasirzadeh Erin van Liemt Sonja Schmer-Galunder 83 8 0 09 Feb 2024
Don't Label Twice: Quantity Beats Quality when Comparing Binary Classifiers on a Budget Florian E. Dorner Moritz Hardt 74 4 0 03 Feb 2024
An Empirical Analysis of Diversity in Argument Summarization Michiel van der Meer Piek T. J. M. Vossen Catholijn M. Jonker P. Murukannaiah 54 8 0 02 Feb 2024
Harm Amplification in Text-to-Image Models Susan Hao Renee Shelby Yuchi Liu Hansa Srinivasan Mukul Bhutani Burcu Karagol Ayan Ryan Poplin Shivani Poddar Sarah Laszlo 98 9 0 01 Feb 2024
A Scoping Study of Evaluation Practices for Responsible AI Tools: Steps Towards Effectiveness Evaluations G. Berman Nitesh Goyal Michael A. Madaio ELM 73 23 0 30 Jan 2024
Diverse, but Divisive: LLMs Can Exaggerate Gender Differences in Opinion Related to Harms of Misinformation Terrence Neumann Sooyong Lee Maria De-Arteaga S. Fazelpour Matthew Lease 78 6 0 29 Jan 2024
Generalized People Diversity: Learning a Human Perception-Aligned Diversity Representation for People Images Hansa Srinivasan Candice Schumann Aradhana Sinha David Madras Gbolahan O. Olanubi Alex Beutel Susanna Ricco Jilin Chen 96 6 0 25 Jan 2024
A Multi-Perspective Machine Learning Approach to Evaluate Police-Driver Interaction in Los Angeles Benjamin A.T. Grahama Lauren Brown Georgios Chochlakis Morteza Dehghani Raquel Delerme ... Mayaguez Salinas Michael Sierra-Arévalo Jackson Trager Nicholas Weller Shrikanth Narayan HAI 89 3 0 24 Jan 2024
Does More Advice Help? The Effects of Second Opinions in AI-Assisted Decision Making Zhuoran Lu Dakuo Wang Ming Yin 75 17 0 13 Jan 2024
Quantifying Divergence for Human-AI Collaboration and Cognitive Trust Muge Kural Ali Gebesçe T. Chubakov Gözde Gül Sahin FedML 43 0 0 14 Dec 2023
On Diversified Preferences of Large Language Model Alignment Dun Zeng Yong Dai Pengyu Cheng Longyue Wang Tianhao Hu Wanshun Chen Nan Du Zenglin Xu ALM 91 17 0 12 Dec 2023
Disentangling Perceptions of Offensiveness: Cultural and Moral Correlates Aida Mostafazadeh Davani Mark Díaz Dylan K. Baker Vinodkumar Prabhakaran AAML 63 18 0 11 Dec 2023
Modeling Uncertainty in Personalized Emotion Prediction with Normalizing Flows P. Milkowski Konrad Karanowski Patryk Wielopolski Jan Kocoñ Przemyslaw Kazienko Maciej Zikeba 102 6 0 10 Dec 2023
Capturing Perspectives of Crowdsourced Annotators in Subjective Learning Tasks Negar Mokhberian Myrl G. Marmarelis F. R. Hopp Valerio Basile Fred Morstatter Kristina Lerman 104 13 0 16 Nov 2023
Subtle Misogyny Detection and Mitigation: An Expert-Annotated Dataset Brooklyn Sheppard Anna Richter Allison Cohen Elizabeth Allyn Smith Tamara Kneese Carolyne Pelletier Ioana Baldini Yue Dong 49 4 0 15 Nov 2023
Human-in-the-loop: Towards Label Embeddings for Measuring Classification Difficulty Katharina Hechinger Christoph Koller Xiao Xiang Zhu Goran Kauermann UQCV 59 1 0 15 Nov 2023
Don't Waste a Single Annotation: Improving Single-Label Classifiers Through Soft Labels Ben Wu Yue Li Yida Mu Carolina Scarton Kalina Bontcheva Xingyi Song 92 12 0 09 Nov 2023
GRASP: A Disagreement Analysis Framework to Assess Group Associations in Perspectives Vinodkumar Prabhakaran Christopher Homan Lora Aroyo Aida Mostafazadeh Davani Alicia Parrish Alex S. Taylor Mark Díaz Ding Wang Greg Serapio-García 97 9 0 09 Nov 2023
A Taxonomy of Rater Disagreements: Surveying Challenges & Opportunities from the Perspective of Annotating Online Toxicity Wenbo Zhang Hangzhi Guo Ian D Kivlichan Vinodkumar Prabhakaran Davis Yadav Amulya Yadav 57 2 0 07 Nov 2023
Architectural Sweet Spots for Modeling Human Label Variation by the Example of Argument Quality: It's Best to Relate Perspectives! Philipp Heinisch Matthias Orlikowski Julia Romberg Philipp Cimiano 44 3 0 06 Nov 2023
The Impact of Preference Agreement in Reinforcement Learning from Human Feedback: A Case Study in Summarization Sian Gooding Hassan Mansoor 42 2 0 02 Nov 2023
Noise Correction on Subjective Datasets Uthman Jinadu Yi Ding 29 1 0 01 Nov 2023
MoCa: Measuring Human-Language Model Alignment on Causal and Moral Judgment Tasks Allen Nie Yuhui Zhang Atharva Amdekar Chris Piech Tatsunori Hashimoto Tobias Gerstenberg 80 40 0 30 Oct 2023