Jury Learning: Integrating Dissenting Voices into Machine Learning Models

7 February 2022

Kayur Patel

Tatsunori Hashimoto

Papers citing "Jury Learning: Integrating Dissenting Voices into Machine Learning Models"

50 / 88 papers shown

Title
Perils of Label Indeterminacy: A Case Study on Prediction of Neurological Recovery After Cardiac Arrest Jakob Schoeffer Maria De-Arteaga Jonathan Elmer 128 0 0 05 Apr 2025
HALO: Human-Aligned End-to-end Image Retargeting with Layered Transformations Yiran Xu Siqi Xie Zhuofang Li Harris Shadmany Yinxiao Li ... Jesse Berent Ming Yang Irfan Essa Jia-Bin Huang Feng Yang VOS 58 0 0 03 Apr 2025
Tasks and Roles in Legal AI: Data Curation, Annotation, and Verification Allison Koenecke Jed Stiglitz David Mimno Matthew Wilkens AILaw ELM 77 0 0 02 Apr 2025
On Benchmarking Human-Like Intelligence in Machines Lance Ying K. M. Collins L. Wong Ilia Sucholutsky Ryan Liu Adrian Weller Tianmin Shu Thomas L. Griffiths Joshua B. Tenenbaum ALM ELM 132 3 0 27 Feb 2025
Exploring the Influence of Label Aggregation on Minority Voices: Implications for Dataset Bias and Model Training Mugdha Pandya Nafise Sadat Moosavi Diana Maynard 79 0 0 05 Dec 2024
Chain of Alignment: Integrating Public Will with Expert Intelligence for Language Model Alignment Andrew Konya Aviv Ovadya K. J. Kevin Feng Quan Ze Chen Lisa Schirch Colin Irwin Amy X. Zhang ALM 44 2 0 15 Nov 2024
LLM-GLOBE: A Benchmark Evaluating the Cultural Values Embedded in LLM Output Elise Karinshak Amanda Hu Kewen Kong Vishwanatha Rao Jingren Wang Jindong Wang Yi Zeng 41 2 0 09 Nov 2024
Harmful YouTube Video Detection: A Taxonomy of Online Harm and MLLMs as Alternative Annotators Claire Jo Miki Wesołowska Magdalena Wojcieszak 30 5 0 06 Nov 2024
Intuitions of Compromise: Utilitarianism vs. Contractualism Jared Moore Yejin Choi Sydney Levine 33 0 0 07 Oct 2024
Policy Prototyping for LLMs: Pluralistic Alignment via Interactive and Collaborative Policymaking K. J. Kevin Feng Inyoung Cheong Quan Ze Chen Amy X. Zhang 44 2 0 13 Sep 2024
User-Driven Value Alignment: Understanding Users' Perceptions and Strategies for Addressing Biased and Discriminatory Statements in AI Companions Xianzhe Fan Qing Xiao Xuhui Zhou Jiaxin Pei Maarten Sap Zhicong Lu Hong Shen 52 6 0 01 Sep 2024
Beyond Preferences in AI Alignment Tan Zhi-Xuan Micah Carroll Matija Franklin Hal Ashton 41 16 0 30 Aug 2024
How will advanced AI systems impact democracy? Christopher Summerfield Lisa Argyle Michiel Bakker Teddy Collins Esin Durmus ... Elizabeth Seger Divya Siddarth Henrik Skaug Sætra MH Tessler M. Botvinick 45 2 0 27 Aug 2024
Improving Context-Aware Preference Modeling for Language Models Silviu Pitis Ziang Xiao Nicolas Le Roux Alessandro Sordoni 38 8 0 20 Jul 2024
Position: Measure Dataset Diversity, Don't Just Claim It Dora Zhao Jerone T. A. Andrews Orestis Papakyriakopoulos Alice Xiang 64 14 0 11 Jul 2024
LEMoN: Label Error Detection using Multimodal Neighbors Haoran Zhang Aparna Balagopalan Nassim Oufattole Hyewon Jeong Yan Wu Jiacheng Zhu Marzyeh Ghassemi 46 0 0 10 Jul 2024
Pareto-Optimal Learning from Preferences with Hidden Context Ryan Boldi Li Ding Lee Spector S. Niekum 67 6 0 21 Jun 2024
Supporting Human Raters with the Detection of Harmful Content using Large Language Models Kurt Thomas Patrick Gage Kelley David Tao Sarah Meiklejohn Owen Vallis Shunwen Tan Blaz Bratanic Felipe Tiengo Ferreira Vijay Eranti Elie Bursztein 38 2 0 18 Jun 2024
Collective Constitutional AI: Aligning a Language Model with Public Input Saffron Huang Divya Siddarth Liane Lovitt Thomas I. Liao Esin Durmus Alex Tamkin Deep Ganguli ELM 59 72 0 12 Jun 2024
Language Model Council: Democratically Benchmarking Foundation Models on Highly Subjective Tasks Justin Zhao Flor Miriam Plaza del Arco A. C. Curry Amanda Cercas Curry ELM ALM 41 1 0 12 Jun 2024
A Taxonomy of Challenges to Curating Fair Datasets Dora Zhao M. Scheuerman Pooja Chitre Jerone T. A. Andrews Georgia Panagiotidou Shawn Walker Kathleen H. Pine Alice Xiang 47 2 0 10 Jun 2024
Counterspeakers' Perspectives: Unveiling Barriers and AI Needs in the Fight against Online Hate Jimin Mun Cathy Buerger Jenny T Liang Joshua Garland Maarten Sap 35 10 0 29 Feb 2024
DISCERN: Designing Decision Support Interfaces to Investigate the Complexities of Workplace Social Decision-Making With Line Managers Pranav Khadpe Lindy Le Kate Nowak Shamsi T. Iqbal Jina Suh 32 7 0 29 Feb 2024
Farsight: Fostering Responsible AI Awareness During AI Application Prototyping Zijie J. Wang Chinmay Kulkarni Lauren Wilcox Michael Terry Michael A. Madaio 40 43 0 23 Feb 2024
Wikibench: Community-Driven Data Curation for AI Evaluation on Wikipedia Tzu-Sheng Kuo Aaron L Halfaker Zirui Cheng Jiwoo Kim Meng-Hsin Wu Tongshuang Wu Kenneth Holstein Haiyi Zhu 62 21 0 21 Feb 2024
A Roadmap to Pluralistic Alignment Taylor Sorensen Jared Moore Jillian R. Fisher Mitchell L. Gordon Niloofar Mireshghallah ... Liwei Jiang Ximing Lu Nouha Dziri Tim Althoff Yejin Choi 65 80 0 07 Feb 2024
Don't Label Twice: Quantity Beats Quality when Comparing Binary Classifiers on a Budget Florian E. Dorner Moritz Hardt 20 4 0 03 Feb 2024
Does More Advice Help? The Effects of Second Opinions in AI-Assisted Decision Making Zhuoran Lu Dakuo Wang Ming Yin 31 16 0 13 Jan 2024
Inclusive Portraits: Race-Aware Human-in-the-Loop Technology Claudia Flores-Saviaga Christopher Curtis Saiph Savage 12 2 0 06 Nov 2023
Modeling subjectivity (by Mimicking Annotator Annotation) in toxic comment identification across diverse communities Senjuti Dutta Sid Mittal Sherol Chen Deepak Ramachandran Ravi Rajakumar Ian D Kivlichan Sunny Mak Alena Butryna Praveen Paritosh University of Tennessee 42 5 0 01 Nov 2023
Eliciting Model Steering Interactions from Users via Data and Visual Design Probes Anamaria Crisan Maddie Shang Eric Brochu 35 3 0 12 Oct 2023
Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations Zhuoyan Li Hangxiao Zhu Zhuoran Lu Ming Yin SyDa 69 67 0 11 Oct 2023
The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values Hannah Rose Kirk Andrew M. Bean Bertie Vidgen Paul Röttger Scott A. Hale ALM 21 41 0 11 Oct 2023
Case Law Grounding: Aligning Judgments of Humans and AI on Socially-Constructed Concepts Quan Ze Chen Amy X. Zhang ELM 56 2 0 10 Oct 2023
The Empty Signifier Problem: Towards Clearer Paradigms for Operationalising "Alignment" in Large Language Models Hannah Rose Kirk Bertie Vidgen Paul Röttger Scott A. Hale 44 2 0 03 Oct 2023
The Participatory Turn in AI Design: Theoretical Foundations and the Current State of Practice Fernando Delgado Stephen Yang Michael A. Madaio Qian Yang 76 100 0 02 Oct 2023
Human Feedback is not Gold Standard Tom Hosking Phil Blunsom Max Bartolo ALM 26 49 0 28 Sep 2023
On the Challenges of Building Datasets for Hate Speech Detection Vitthal Bhandari 18 1 0 06 Sep 2023
How Crowd Worker Factors Influence Subjective Annotations: A Study of Tagging Misogynistic Hate Speech in Tweets Danula Hettiachchi I. Holcombe-James Stephanie Livingstone Anjalee de Silva Matthew Lease Flora D. Salim Mark Sanderson 18 10 0 03 Sep 2023
Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Duties Taylor Sorensen Liwei Jiang Jena D. Hwang Sydney Levine Valentina Pyatkin ... Kavel Rao Chandra Bhagavatula Maarten Sap J. Tasioulas Yejin Choi SLR 18 51 0 02 Sep 2023
Cura: Curation at Social Media Scale Wanrong He Mitchell L. Gordon Lindsay Popowski Michael S. Bernstein 9 10 0 26 Aug 2023
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback Stephen Casper Xander Davies Claudia Shi T. Gilbert Jérémy Scheurer ... Erdem Biyik Anca Dragan David M. Krueger Dorsa Sadigh Dylan Hadfield-Menell ALM OffRL 47 472 0 27 Jul 2023
Mitigating Voter Attribute Bias for Fair Opinion Aggregation Ryosuke Ueda Koh Takeuchi H. Kashima 8 1 0 20 Jul 2023
Conformal prediction under ambiguous ground truth David Stutz Abhijit Guha Roy Tatiana Matejovicova Patricia Strachan A. Cemgil Arnaud Doucet 31 15 0 18 Jul 2023
Subjective Crowd Disagreements for Subjective Data: Uncovering Meaningful CrowdOpinion with Population-level Learning Tharindu Cyril Weerasooriya Sarah K. K. Luger Saloni Poddar Ashiqur R. KhudaBukhsh Christopher Homan 15 5 0 07 Jul 2023
Scaling Laws Do Not Scale Fernando Diaz Michael A. Madaio 23 8 0 05 Jul 2023
Evaluating AI systems under uncertain ground truth: a case study in dermatology David Stutz A. Cemgil Abhijit Guha Roy Tatiana Matejovicova Melih Barsbey ... Yossi Matias Pushmeet Kohli Yun-hui Liu Arnaud Doucet Alan Karthikesalingam 30 4 0 05 Jul 2023
Towards Measuring the Representation of Subjective Global Opinions in Language Models Esin Durmus Karina Nyugen Thomas I. Liao Nicholas Schiefer Amanda Askell ... Alex Tamkin Janel Thamkul Jared Kaplan Jack Clark Deep Ganguli 35 207 0 28 Jun 2023
The Ecological Fallacy in Annotation: Modelling Human Label Variation goes beyond Sociodemographics Matthias Orlikowski Paul Röttger Philipp Cimiano Italy 26 26 0 20 Jun 2023
Designing Closed-Loop Models for Task Allocation Vijay Keswani L. E. Celis K. Kenthapadi Matthew Lease 16 0 0 31 May 2023