CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation

24 October 2023

Taiwei Shi

Nancy F. Chen

Diyi Yang

Papers citing "CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation"

50 / 51 papers shown

Title
ViMRHP: A Vietnamese Benchmark Dataset for Multimodal Review Helpfulness Prediction via Human-AI Collaborative Annotation T. Nguyen D. Nguyen Son T. Luu Kiet Van Nguyen 11 0 0 12 May 2025
AI Chatbots for Mental Health: Values and Harms from Lived Experiences of Depression Dong Whi Yoo Jiayue Melissa Shi Violeta J. Rodriguez Koustuv Saha AI4MH 46 0 0 26 Apr 2025
Can Third-parties Read Our Emotions? Jiayi Li Yingfan Zhou Pranav Narayanan Venkit Halima Binte Islam Sneha Arya Shomir Wilson Sarah Rajtmajer 36 0 0 25 Apr 2025
A Survey of Large Language Models in Mental Health Disorder Detection on Social Media Zhuohan Ge Nicole Hu Darian Li Yubo Wang Shihao Qi Yuming Xu Han Shi J. Zhang AI4MH 54 0 0 03 Apr 2025
Brains vs. Bytes: Evaluating LLM Proficiency in Olympiad Mathematics Hamed Mahdavi Alireza Hashemi Majid Daliri Pegah Mohammadipour Alireza Farhadi Samira Malek Yekta Yazdanifard Amir Khasahmadi V. Honavar ELM LRM 49 1 0 01 Apr 2025
Evaluating how LLM annotations represent diverse views on contentious topics Megan A. Brown Shubham Atreja Libby Hemphill Patrick Y. Wu 46 0 0 29 Mar 2025
SynGraph: A Dynamic Graph-LLM Synthesis Framework for Sparse Streaming User Sentiment Modeling Xin Zhang Qiyu Wei Yingjie Zhu L. Zhang Deyu Zhou Sophia Ananiadou 43 0 0 06 Mar 2025
SEOE: A Scalable and Reliable Semantic Evaluation Framework for Open Domain Event Detection Yi-Fan Lu Xian-Ling Mao Tian Lan Tong Zhang Yu-Shi Zhu Heyan Huang 47 0 0 05 Mar 2025
Few-shot LLM Synthetic Data with Distribution Matching Jiyuan Ren Zhaocheng Du Zhihao Wen Qinglin Jia Sunhao Dai Chuhan Wu Zhenhua Dong SyDa 75 0 0 09 Feb 2025
Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedback Lester James Validad Miranda Yizhong Wang Yanai Elazar Sachin Kumar Valentina Pyatkin Faeze Brahman Noah A. Smith Hannaneh Hajishirzi Pradeep Dasigi 45 8 0 08 Jan 2025
Automated Collection of Evaluation Dataset for Semantic Search in Low-Resource Domain Language Anastasia Zhukova Christian E. Matt Bela Gipp 77 2 0 13 Dec 2024
Are LLMs Better than Reported? Detecting Label Errors and Mitigating Their Effect on Model Performance Omer Nahum Nitay Calderon Orgad Keller Idan Szpektor Roi Reichart 23 1 0 24 Oct 2024
Uncovering the Internet's Hidden Values: An Empirical Study of Desirable Behavior Using Highly-Upvoted Content on Reddit Agam Goyal Charlotte Lambert Eshwar Chandrasekharan 23 2 0 16 Oct 2024
A Survey on Data Synthesis and Augmentation for Large Language Models Ke Wang Jiahui Zhu Minjie Ren Z. Liu Shiwei Li ... Chenkai Zhang Xiaoyu Wu Qiqi Zhan Qingjie Liu Yunhong Wang SyDa 36 15 0 16 Oct 2024
Mitigating Selection Bias with Node Pruning and Auxiliary Options Hyeong Kyu Choi Weijie Xu Chi Xue Stephanie Eckman Chandan K. Reddy 26 1 0 27 Sep 2024
LLM-as-a-Judge & Reward Model: What They Can and Cannot Do Guijin Son Hyunwoo Ko Hoyoung Lee Yewon Kim Seunghyeok Hong ALM ELM 46 5 0 17 Sep 2024
Model-in-the-Loop (MILO): Accelerating Multimodal AI Data Annotation with LLMs Yifan Wang David Stevens Pranay Shah Wenwen Jiang Miao Liu ... Boying Gong Daniel Lee Jiabo Hu Ning Zhang Bob Kamma 30 1 0 16 Sep 2024
Keeping Humans in the Loop: Human-Centered Automated Annotation with Generative AI Nicholas Pangakis Samuel Wolken 21 3 0 14 Sep 2024
ConsistencyTrack: A Robust Multi-Object Tracker with a Generation Strategy of Consistency Model Lifan Jiang Zhihui Wang Siqi Yin Guangxiao Ma Peng Zhang Boxi Wu DiffM 51 0 0 28 Aug 2024
Can Unconfident LLM Annotations Be Used for Confident Conclusions? Kristina Gligorić Tijana Zrnic Cinoo Lee Emmanuel J. Candès Dan Jurafsky 63 4 0 27 Aug 2024
Automating Knowledge Discovery from Scientific Literature via LLMs: A Dual-Agent Approach with Progressive Ontology Prompting Yuting Hu Dancheng Liu Qingyun Wang Charles Yu Heng Ji Jinjun Xiong LLMAG 30 0 0 20 Aug 2024
VolDoGer: LLM-assisted Datasets for Domain Generalization in Vision-Language Tasks Juhwan Choi Junehyoung Kwon Jungmin Yun Seunguk Yu Youngbin Kim 36 1 0 29 Jul 2024
MMM: Multilingual Mutual Reinforcement Effect Mix Datasets & Test with Open-domain Information Extraction Large Language Models Chengguang Gan Qingyu Yin Xinyang He Hanjun Wei Yunhao Liang ... Shijian Wang Hexiang Huang Qinghao Zhang Shiwen Ni Tatsunori Mori 27 0 0 15 Jul 2024
Real-time Speech Summarization for Medical Conversations Khai Le-Duc Khai-Nguyen Nguyen Long Vo-Dang Truong Son-Hy MedIm 49 1 0 22 Jun 2024
OATH-Frames: Characterizing Online Attitudes Towards Homelessness with LLM Assistants Jaspreet Ranjit Brihi Joshi Rebecca Dorn Laura Petry Olga Koumoundouros Jayne Bottarini Peichen Liu Eric Rice Swabha Swayamdipta 22 1 0 21 Jun 2024
Decoding the Narratives: Analyzing Personal Drug Experiences Shared on Reddit Layla A. Bouzoubaa Elham Aghakhani Max Song Minh Trinh R. Rezapour 18 1 0 17 Jun 2024
On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey Lin Long Rui Wang Ruixuan Xiao Junbo Zhao Xiao Ding Gang Chen Haobo Wang SyDa 51 88 0 14 Jun 2024
Silent Signals, Loud Impact: LLMs for Word-Sense Disambiguation of Coded Dog Whistles Julia Kruk Michela Marchini Rijul Magu Caleb Ziems D. Muchlinski Diyi Yang 22 1 0 10 Jun 2024
mCSQA: Multilingual Commonsense Reasoning Dataset with Unified Creation Strategy by Language Models and Humans Yusuke Sakai Hidetaka Kamigaito Taro Watanabe LRM 38 2 0 06 Jun 2024
Selective Annotation via Data Allocation: These Data Should Be Triaged to Experts for Annotation Rather Than the Model Chen Huang Yang Deng Wenqiang Lei Jiancheng Lv Ido Dagan 30 4 0 20 May 2024
CrossIn: An Efficient Instruction Tuning Approach for Cross-Lingual Knowledge Alignment Geyu Lin Bin Wang Zhengyuan Liu Nancy F. Chen 32 7 0 18 Apr 2024
Context Does Matter: Implications for Crowdsourced Evaluation Labels in Task-Oriented Dialogue Systems Clemencia Siro Mohammad Aliannejadi Maarten de Rijke 19 3 0 15 Apr 2024
Scaffolding Language Learning via Multi-modal Tutoring Systems with Pedagogical Instructions Zhengyuan Liu Stella Xin Yin Carolyn Lee Nancy F. Chen AI4Ed 25 11 0 04 Apr 2024
LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancement Nicholas Lee Thanakul Wattanawong Sehoon Kim K. Mangalam Sheng Shen Gopala Anumanchipalli Michael W. Mahoney Kurt Keutzer A. Gholami 58 46 0 22 Mar 2024
Multi-Level Feedback Generation with Large Language Models for Empowering Novice Peer Counselors Alicja Chaszczewicz Raj Sanjay Shah Ryan Louie B. Arnow Robert E. Kraut Diyi Yang OffRL 14 9 0 21 Mar 2024
Fine-grainedly Synthesize Streaming Data Based On Large Language Models With Graph Structure Understanding For Data Sparsity Xin Zhang Linhai Zhang Deyu Zhou Guoqiang Xu SyDa 15 0 0 10 Mar 2024
Data Augmentation using Large Language Models: Data Perspectives, Learning Paradigms and Challenges Bosheng Ding Chengwei Qin Ruochen Zhao Tianze Luo Xinze Li Guizhen Chen Wenhan Xia Junjie Hu A. Luu Shafiq R. Joty 29 18 0 05 Mar 2024
Social Intelligence Data Infrastructure: Structuring the Present and Navigating the Future Minzhi Li Weiyan Shi Caleb Ziems Diyi Yang 20 8 0 28 Feb 2024
Cost-Efficient Subjective Task Annotation and Modeling through Few-Shot Annotator Adaptation Preni Golazizian Ali Omrani Alireza S. Ziabari Morteza Dehghani 16 1 0 21 Feb 2024
ARL2: Aligning Retrievers for Black-box Large Language Models via Self-guided Adaptive Relevance Labeling Lingxi Zhang Yue Yu Kuan-Chieh Jackson Wang Chao Zhang VLM RALM 22 4 0 21 Feb 2024
Large Language Models for Data Annotation: A Survey Zhen Tan Dawei Li Song Wang Alimohammad Beigi Bohan Jiang Amrita Bhattacharjee Mansooreh Karami Jundong Li Lu Cheng Huan Liu SyDa 42 44 0 21 Feb 2024
GPTs Are Multilingual Annotators for Sequence Generation Tasks Juhwan Choi Eunju Lee Kyohoon Jin Youngbin Kim 25 10 0 08 Feb 2024
HR-MultiWOZ: A Task Oriented Dialogue (TOD) Dataset for HR LLM Agent Weijie Xu Zicheng Huang Wenxiang Hu Xi Fang Rajesh Kumar Cherukuri Naumaan Nayyar Lorenzo Malandri Srinivasan H. Sengamedu 11 5 0 01 Feb 2024
I Think, Therefore I am: Benchmarking Awareness of Large Language Models Using AwareBench Yuan Li Yue Huang Yuli Lin Siyuan Wu Yao Wan Lichao Sun LLMAG ELM 38 4 0 31 Jan 2024
DecipherPref: Analyzing Influential Factors in Human Preference Judgments via GPT-4 Ye Hu Kaiqiang Song Sangwoo Cho Xiaoyang Wang H. Foroosh Fei Liu 13 11 0 24 May 2023
Distill or Annotate? Cost-Efficient Fine-Tuning of Compact Models Junmo Kang Wei-ping Xu Alan Ritter 33 15 0 02 May 2023
Will Affective Computing Emerge from Foundation Models and General AI? A First Evaluation on ChatGPT Mostafa M. Amin Erik Cambria Björn W. Schuller AI4MH 52 70 0 03 Mar 2023
TempoWiC: An Evaluation Benchmark for Detecting Meaning Shift in Social Media Daniel Loureiro Aminette D'Souza Areej Muhajab Isabella A. White Gabriel Wong Luis Espinosa Anke Leonardo Neves Francesco Barbieri Jose Camacho-Collados 27 25 0 15 Sep 2022
Self-Consistency Improves Chain of Thought Reasoning in Language Models Xuezhi Wang Jason W. Wei Dale Schuurmans Quoc Le Ed H. Chi Sharan Narang Aakanksha Chowdhery Denny Zhou ReLM BDL LRM AI4CE 297 3,163 0 21 Mar 2022
SynthBio: A Case Study in Human-AI Collaborative Curation of Text Datasets Ann Yuan Daphne Ippolito Vitaly Nikolaev Chris Callison-Burch Andy Coenen Sebastian Gehrmann SyDa 104 20 0 11 Nov 2021