Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Tasks

20 April 2023

Papers citing "Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Tasks"

50 / 76 papers shown

Title
Embracing Diversity: A Multi-Perspective Approach with Soft Labels Benedetta Muscato Praveen Bushipaka Gizem Gezici Lucia Passaro F. Giannotti Tommaso Cucinotta 34 0 0 01 Mar 2025
Think Together and Work Better: Combining Humans' and LLMs' Think-Aloud Outcomes for Effective Text Evaluation SeongYeub Chu JongWoo Kim MunYong Yi 55 1 0 21 Feb 2025
A Large-Scale Benchmark for Vietnamese Sentence Paraphrases Sang Quang Nguyen Kiet Van Nguyen 60 0 0 11 Feb 2025
Scaling Public Health Text Annotation: Zero-Shot Learning vs. Crowdsourcing for Improved Efficiency and Labeling Accuracy Kamyar Kazari Yong Chen Zahra Shakeri AI4MH 53 0 0 10 Feb 2025
Can LLMs Rank the Harmfulness of Smaller LLMs? We are Not There Yet Berk Atil Vipul Gupta Sarkar Snigdha Sarathi Das R. Passonneau 115 0 0 07 Feb 2025
SkillScope: A Tool to Predict Fine-Grained Skills Needed to Solve Issues on GitHub Benjamin C. Carter Jonathan Rivas Contreras Carlos A. Llanes Villegas Pawan Acharya Jack Utzerath ... Dylan Johnson Jacob Penney Igor Steinmacher M. Gerosa Fabio Santos 37 0 0 28 Jan 2025
Automated Collection of Evaluation Dataset for Semantic Search in Low-Resource Domain Language Anastasia Zhukova Christian E. Matt Bela Gipp 79 2 0 13 Dec 2024
Human-LLM Hybrid Text Answer Aggregation for Crowd Annotations Jiyi Li 27 1 0 22 Oct 2024
Towards Hybrid Intelligence in Journalism: Findings and Lessons Learnt from a Collaborative Analysis of Greek Political Rhetoric by ChatGPT and Humans Thanasis Troboukis Kelly Kiki Antonis Galanopoulos Pavlos Sermpezis Stelios Karamanidis Ilias Dimitriadis Athena Vakali 21 0 0 17 Oct 2024
LLM Chain Ensembles for Scalable and Accurate Data Annotation David Farr Nico Manzonelli Iain Cruickshank Kate Starbird Jevin West 29 3 0 16 Oct 2024
A Survey on Data Synthesis and Augmentation for Large Language Models Ke Wang Jiahui Zhu Minjie Ren Z. Liu Shiwei Li ... Chenkai Zhang Xiaoyu Wu Qiqi Zhan Qingjie Liu Yunhong Wang SyDa 38 15 0 16 Oct 2024
Are Expert-Level Language Models Expert-Level Annotators? Yu-Min Tseng Wei-Lin Chen Chung-Chi Chen Hsin-Hsi Chen ALM 34 0 0 04 Oct 2024
MedHalu: Hallucinations in Responses to Healthcare Queries by Large Language Models Vibhor Agarwal Yiqiao Jin Mohit Chandra Munmun De Choudhury Srijan Kumar Nishanth R. Sastry HILM LM&MA 42 4 0 29 Sep 2024
Semantics Preserving Emoji Recommendation with Large Language Models Zhongyi Qiu Kangyi Qiu Hanjia Lyu Wei Xiong Jiebo Luo 39 0 0 16 Sep 2024
Keeping Humans in the Loop: Human-Centered Automated Annotation with Generative AI Nicholas Pangakis Samuel Wolken 26 3 0 14 Sep 2024
RED-CT: A Systems Design Methodology for Using LLM-labeled Data to Train and Deploy Edge Classifiers for Computational Social Science David Farr Nico Manzonelli Iain Cruickshank Jevin West 28 1 0 15 Aug 2024
CodeMirage: Hallucinations in Code Generated by Large Language Models Vibhor Agarwal Yulong Pei Salwa Alamir Xiaomo Liu 40 4 0 14 Aug 2024
Representation Bias in Political Sample Simulations with Large Language Models Weihong Qi Hanjia Lyu Jiebo Luo 31 5 0 16 Jul 2024
Knowledge Distillation in Automated Annotation: Supervised Text Classification with LLM-Generated Training Labels Nicholas Pangakis Samuel Wolken 27 15 0 25 Jun 2024
Investigating Annotator Bias in Large Language Models for Hate Speech Detection Amit Das Zheng Zhang Fatemeh Jamshidi Vinija Jain Aman Chadha Nilanjana Raychawdhary Mary J. Sandage Lauramarie Pope Gerry V. Dozier Cheryl Seals 19 2 0 17 Jun 2024
Advancing Annotation of Stance in Social Media Posts: A Comparative Analysis of Large Language Models and Crowd Sourcing Mao Li Frederick Conrad 29 1 0 11 Jun 2024
Organic Data-Driven Approach for Turkish Grammatical Error Correction and LLMs Asim Ersoy O. T. Yildiz 28 0 0 24 May 2024
On the Brittle Foundations of ReAct Prompting for Agentic Large Language Models Mudit Verma Siddhant Bhambri Subbarao Kambhampati ReLM LRM 48 8 0 22 May 2024
Aggregation of Reasoning: A Hierarchical Framework for Enhancing Answer Selection in Large Language Models Zhangyue Yin Qiushi Sun Qipeng Guo Zhiyuan Zeng Xiaonan Li ... Qinyuan Cheng Ding Wang Xiaofeng Mou Xipeng Qiu XuanJing Huang LRM 41 3 0 21 May 2024
The Effectiveness of LLMs as Annotators: A Comparative Overview and Empirical Analysis of Direct Representation Maja Pavlovic Massimo Poesio 32 17 0 02 May 2024
ChatGPT Rates Natural Language Explanation Quality Like Humans: But on Which Scales? Fan Huang Haewoon Kwak Kunwoo Park Jisun An ALM ELM AI4MH 24 12 0 26 Mar 2024
Towards Interpretable Hate Speech Detection using Large Language Model-extracted Rationales Ayushi Nirmal Amrita Bhattacharjee Paras Sheth Huan Liu AAML 20 11 0 19 Mar 2024
Harnessing Artificial Intelligence to Combat Online Hate: Exploring the Challenges and Opportunities of Large Language Models in Hate Speech Detection Tharindu Kumarage Amrita Bhattacharjee Joshua Garland 39 7 0 12 Mar 2024
Generating Hard-Negative Out-of-Scope Data with ChatGPT for Intent Classification Zhijian Li Stefan Larson Kevin Leach OODD 21 1 0 08 Mar 2024
OffensiveLang: A Community Based Implicit Offensive Language Dataset Amit Das Mostafa Rahgouy Dongji Feng Zheng Zhang Tathagata Bhattacharya ... Aman Chadha Mary J. Sandage Lauramarie Pope Gerry V. Dozier Cheryl Seals 16 1 0 04 Mar 2024
MEGAnno+: A Human-LLM Collaborative Annotation System H. Kim Kushan Mitra Rafael Li Chen Sajjadur Rahman Dan Zhang 33 23 0 28 Feb 2024
If in a Crowdsourced Data Annotation Pipeline, a GPT-4 Zeyu He Huang Chieh-Yang C. C. Ding Shaurya Rohatgi Ting-Hao 'Kenneth' Huang 22 30 0 26 Feb 2024
EvoGrad: A Dynamic Take on the Winograd Schema Challenge with Human Adversaries Jing Han Sun Ali Emami 27 3 0 20 Feb 2024
AFaCTA: Assisting the Annotation of Factual Claim Detection with Reliable LLM Annotators Jingwei Ni Minjing Shi Dominik Stammbach Mrinmaya Sachan Elliott Ash Markus Leippold HILM 10 11 0 16 Feb 2024
Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs Simone Balloccu Patrícia Schmidtová Mateusz Lango Ondrej Dusek SILM ELM PILM 21 155 0 06 Feb 2024
APT-Pipe: A Prompt-Tuning Tool for Social Data Annotation using ChatGPT Yiming Zhu Zhizhuo Yin Gareth Tyson Ehsan-ul Haq Lik-Hang Lee Pan Hui ALM 33 6 0 24 Jan 2024
Adaptive Crowdsourcing Via Self-Supervised Learning Anmol Kagrecha Henrik Marklund Benjamin Van Roy Hong Jun Jeon Richard Zeckhauser FedML 13 0 0 24 Jan 2024
A Comparative Study on Annotation Quality of Crowdsourcing and LLM via Label Aggregation Jiyi Li 26 16 0 18 Jan 2024
DepressionEmo: A novel dataset for multilabel classification of depression emotions Abu Bakar Siddiqur Rahman Hoang-Thang Ta Lotfollah Najjar A. Azadmanesh A. Gonul AI4MH 28 10 0 09 Jan 2024
The Butterfly Effect of Altering Prompts: How Small Changes and Jailbreaks Affect Large Language Model Performance A. Salinas Fred Morstatter 30 49 0 08 Jan 2024
Evaluation is all you need. Prompting Generative Large Language Models for Annotation Tasks in the Social Sciences. A Primer using Open Models Maximilian Weber Merle Reichardt 71 9 0 30 Dec 2023
Understanding News Creation Intents: Frame, Dataset, and Method Zhengjia Wang Danding Wang Qiang Sheng Juan Cao Silong Su Yifan Sun Beizhe Hu Siyuan Ma 10 4 0 27 Dec 2023
Let the LLMs Talk: Simulating Human-to-Human Conversational QA via Zero-Shot LLM-to-LLM Interactions Zahra Abbasiantaeb Yifei Yuan Evangelos Kanoulas Mohammad Aliannejadi 17 54 0 05 Dec 2023
Complementary Advantages of ChatGPTs and Human Readers in Reasoning: Evidence from English Text Reading Comprehension Tongquan Zhou Yao Zhang Siyi Cao Yulu Li Tao Wang AI4MH LRM 24 2 0 17 Nov 2023
Large Language Models are Few-Shot Training Example Generators: A Case Study in Fallacy Recognition Tariq Alhindi Smaranda Muresan Preslav Nakov HILM LRM 38 4 0 16 Nov 2023
Interactive Multi-fidelity Learning for Cost-effective Adaptation of Language Model with Sparse Human Supervision Jiaxin Zhang Zhuohang Li Kamalika Das Kumar Sricharan 23 2 0 31 Oct 2023
HateRephrase: Zero- and Few-Shot Reduction of Hate Intensity in Online Posts using Large Language Models Vibhor Agarwal Yu Chen Nishanth R. Sastry 14 6 0 21 Oct 2023
Probing LLMs for hate speech detection: strengths and vulnerabilities Sarthak Roy Ashish Harshavardhan Animesh Mukherjee Punyajoy Saha 63 32 0 19 Oct 2023
Automated Claim Matching with Large Language Models: Empowering Fact-Checkers in the Fight Against Misinformation Eun Cheol Choi Emilio Ferrara 21 17 0 13 Oct 2023
Multiclass Classification of Policy Documents with Large Language Models Erkan Gunes Christoffer Florczak 18 1 0 12 Oct 2023