The Ecological Fallacy in Annotation: Modelling Human Label Variation goes beyond Sociodemographics

20 June 2023

Paul Röttger

Papers citing "The Ecological Fallacy in Annotation: Modelling Human Label Variation goes beyond Sociodemographics"

20 / 20 papers shown

Title
Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals' Subjective Text Perceptions Matthias Orlikowski Jiaxin Pei Paul Röttger Philipp Cimiano David Jurgens Dirk Hovy 54 1 0 28 Feb 2025
SubData: A Python Library to Collect and Combine Datasets for Evaluating LLM Alignment on Downstream Tasks Leon Fröhling Pietro Bernardelle Gianluca Demartini ALM 74 0 0 21 Dec 2024
Accurate and Data-Efficient Toxicity Prediction when Annotators Disagree Harbani Jaggi Kashyap Murali Eve Fleisig Erdem Bıyık 22 1 0 16 Oct 2024
Personas with Attitudes: Controlling LLMs for Diverse Data Annotation Leon Fröhling Gianluca Demartini Dennis Assenmacher 24 5 0 15 Oct 2024
Re-examining Sexism and Misogyny Classification with Annotator Attitudes Aiqi Jiang Nikolas Vitsakis Tanvi Dinkar Gavin Abercrombie Ioannis Konstas 37 1 0 04 Oct 2024
Hate Personified: Investigating the role of LLMs in content moderation Sarah Masud Sahajpreet Singh Viktor Hangya Alexander M. Fraser Tanmoy Chakraborty 17 7 0 03 Oct 2024
Let Guidelines Guide You: A Prescriptive Guideline-Centered Data Annotation Methodology Federico Ruggeri Eleonora Misino Arianna Muti Katerina Korre Paolo Torroni Alberto Barrón-Cedeño 19 0 0 20 Jun 2024
Annotation alignment: Comparing LLM and human annotations of conversational safety Rajiv Movva Pang Wei Koh Emma Pierson ALM 37 3 0 10 Jun 2024
Designing NLP Systems That Adapt to Diverse Worldviews Claudiu Creanga Liviu P. Dinu 13 0 0 18 May 2024
The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels Eve Fleisig Su Lin Blodgett Dan Klein Zeerak Talat 25 13 0 09 May 2024
Annotator-Centric Active Learning for Subjective NLP Tasks Michiel van der Meer Neele Falk P. Murukannaiah Enrico Liscio 27 8 0 24 Apr 2024
D3CODE: Disentangling Disagreements in Data across Cultures on Offensiveness Detection and Evaluation Aida Mostafazadeh Davani Mark Díaz Dylan K. Baker Vinodkumar Prabhakaran 19 8 0 16 Apr 2024
Emotion Analysis in NLP: Trends, Gaps and Roadmap for Future Directions Flor Miriam Plaza del Arco Alba Curry A. C. Curry Dirk Hovy 33 14 0 02 Mar 2024
Quantifying the Persona Effect in LLM Simulations Tiancheng Hu Nigel Collier 16 49 0 16 Feb 2024
Comparing Pre-trained Human Language Models: Is it Better with Human Context as Groups, Individual Traits, or Both? Nikita Soni Niranjan Balasubramanian H. A. Schwartz Dirk Hovy 22 3 0 23 Jan 2024
Capturing Perspectives of Crowdsourced Annotators in Subjective Learning Tasks Negar Mokhberian Myrl G. Marmarelis F. R. Hopp Valerio Basile Fred Morstatter Kristina Lerman 22 9 0 16 Nov 2023
GRASP: A Disagreement Analysis Framework to Assess Group Associations in Perspectives Vinodkumar Prabhakaran Christopher Homan Lora Aroyo Aida Mostafazadeh Davani Alicia Parrish Alex S. Taylor Mark Díaz Ding Wang Greg Serapio-García 29 9 0 09 Nov 2023
Architectural Sweet Spots for Modeling Human Label Variation by the Example of Argument Quality: It's Best to Relate Perspectives! Philipp Heinisch Matthias Orlikowski Julia Romberg Philipp Cimiano 8 3 0 06 Nov 2023
Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Prompting Tilman Beck Hendrik Schuff Anne Lauscher Iryna Gurevych 15 32 0 13 Sep 2023
When Do Annotator Demographics Matter? Measuring the Influence of Annotator Demographics with the POPQUORN Dataset Jiaxin Pei David Jurgens 14 31 0 12 Jun 2023